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FOREWORD 

One of the research goals of the Decision Sciences Laboratory 
is the development of design principles for automated training sub- 
systems which could be built into future Information Systems. Such 
subsystems would provide Information Systems with the capability of 
automatically training their own operators. The need for such on- 
the-job training capability has already become apparent. To be able 
to design such a capability requires first the solution of many con- 
ceptual and experimental problems. 

Task 765204, Automated Training for Information Systems, under 
Project 7682, Man-Computer Information Processing, was established to 
formulate and answer some of these questions. 

This report is one in a planned series supporting Task 768204. 
The project was undertaken by Decision Sciences Laboratory in support 
of the 473L Systems Program Office. Dr. Sylvia R. Mayer of Decision 
Sciences Laboratory served as Air Force Task Scientist and contract 
monitor. 

Contract support was provided by the American Institutes for 
Research, Pittsburgh, Pennsylvania, under Contract No. AF 19(628)-2935^ 
with Mr. Harris H. Shettel, principal investigator; Mrs. Doris Clapp 
Slough, project director; Mr. David P. Yens; and Miss Judi L. Northrup. 

The technical guidance and support provided by Lt . Colonel Wood 
Ellis of the USAF Command Post are gratefully acknowledged. Thanks 
are also extended to the personnel of the USAF Command Post who served 
in the field tests of this course. 

In ensuring the development of the operational specification for 
computer directed training in accordance with approved 473L standaros, 
the A.I.R. project staff had the extremely valuable guidance of 
Dr. William F. Bennett, Mr. Marvin L. Chenevert, and Mr. Jack D. 
Schiff of the Federal Systems Division of International Business 
Machines, Inc. This is gratefully acknowledged. In addition, helpful 
contributions to the training design for the computer directed training 
package were made by Lt . Colonel Wood Ellis, Dr. Sylvia R. Mayer, 
Mr. Jack D. Schiff, and Mr. Marvin L. Chenevert. 

This technical report has been reviewed and is approved . 

DONALD W. CONNOLLY / ROY MORGAN 

Project Officer / Colonel, USAF 

Decision Sciences Laboratory Director, Decision Sciences Laboratory 
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ABSTRACT 



This report summarizes the development and evaluation of a 
programed, self -Instructional course for on-the-job training of 
Air Staff personnel In the use of Intermediate Query Language, 
Model II, This Is an Information retrieval language used with 
the computer based, Air Force command and control system. System 
473L. In addition, it describes a computer directed training 
capability that was designed specifically to use System 473L Itself 
to effectively and efficiently provide training In Query Language, 
The report describes the need for on-the-job training and the 
rationale for a computer directed training capability to provide 
this training. It describes the development of the programed text, 
^-he text Itself, and the effectiveness of the text materials based 
on tryout data. Finally, a description of the proposed computer 
directed training course Is given, with emphasis on the training 
design. The 473L System configuration using the AN/FYQ-11 computer, 
towards which this study was oriented, will not be Implemented for 
the Headquarters U, S. Air Force Command and Control System. How- 
ever, this design study for the training subsystem may be of Inter- 
est to researchers on computer-directed Instructional systems • 
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Editor's Note 



The reader should note that the 473L System configuration 
using the AN/FYQ-11 computer, towards which this study was ori- 
ented^ will not be implemented for the Headquarters U. S. Air 
Force Command and Control System. 

This report is presented since the general design of the 
training subsystem could serve as a model for other computer- 
directed training courses in other computer-based military infor- 
mation systems. 
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Section I 
INTRODUCTION 



1.1 BACKGROUND 



Query Language is a constrained version of the English 
language which has been developed as a mode of man-machine 
communication in the Air Force Command and Control System, 
System 473L (this system is described in section 1.2). Query 
Language has evolved over a period of time to accommodate 
projected changes in the system hardware (viz., from the IBM 
l4lO to the Librascope 3055)^ and to permit problem solutions 
for additional areas of resource management. This evolution 
involved three successive versions of the language, in the fol- 
lowing order: OTC Query Language; Query Language Model I; 
and Query Language Model II. The modifications in the language 
were substantial enough so that personnel trained in the use 
of the Query Language designed for the Operational Training 
Capability (OTC) phase of System 473L could not easily transi- 
tion to the use of the Query Language designed for the Model II 
phase of System 473L. In addition, appropriate training was 
needed for new personnel. Therefore, a new training package 
was requested to teach the version of Query Language appropriate 
to the Model II phase of System 473L. It was further requested 
that a detailed description be developed for a computer directed 
training capability, in accord with 473L standards, so that the 
Air Force could evaluate the feasibility of implementing such a 
capability. 

This report describes the development of a programed, self- 
instructional course designed for on-the-job training of System 
473L users in Intermediate Query Language, Model II. "^ In addi- 
tion, it describes the development of an onerational specifica- 
tion for a computer directed training (CDT) capability designed 
specifically to use the 473L System itself to effectively and 
efficiently train Air Staff personnel in the use of Intermediate 
Query Language, Model II. The proposed computer directed 
training capability has further significance in its potential, 
with appropriate adaptations, for providing training in other 
uses of System 473L. In addition, the general design of the 
training course set forth in the operational specification, 
i.e., the training sequence logic, could serve as a model for 
similar computer directed training courses in other computer 
based military systems. 

1,2 THE AIR FORCE COMMAND AND CONTROL SYSTEM (473L) 

System 473L is an information processing and retrieval 
system located in the Air Force Command Post at the Pentagon. 
An integral part of the system is a large capacity computer. 



^The scope of Intermediate Query Language, Model II, is defined 
in Chapter 3 of 473L-OS-40: Operat ional Specification fo r 
Querjr^ L anguage, Mo del II, dated 13'April '19557 Unclassified. 



The system is designed for use by Air Staff personnel In 
solving USAF resource management problems. There are two 
basic methods of communication between the System 473L 
operator and the data processing subsystem: l) operational capability 
overlays and 2) Query Language. With the overlay capabilities 
the operator is guided in selecting and making his input but 
his retrieval of Information is restricted to a set of pre- 
viously specified outputs. With Query Language, the operator's 
Input is not guided, but his retrieval of information is not as 
restricted. Thus, Query Language is a vital complement to the 
overlay method of communication with System 473L. 

1.3 THE TRAINING PROBLEM 

In developing an overall training strategy, consideration 
was given to two interacting aspects of the training problem: 
the training task Itself and the trainees. 

The use of System 473L is not restricted to a small number 
of system operators ; it is intended that this system, including 
Query Language, be used by a large number of Air Staff personnel. 
These constituted the potential trainees for this course. 

It was specified that the training course should teach 
only Intermediate Query Language, since it Includes those 
aspects of the language that are most commonly used by Air 
Staff personnel. The more advanced uses of Query Language are 
restricted primarily to programmer use (e.g., for maintaining 
and updating the data base), and thus these uses were not con- 
sidered appropriate for the proposed course. 

Proficient use of Intermediate Query Language requires a 
knowledge of both the structure and contents of the system 
data base and the rules regulating the use of vocabulary, 
syntax, grammar, and punctuation of Query Language. In addi- 
tion, since Query Language must be entered on the System 473L 
Integrated console, the user must ultimately become proficient 
in the operation of this console. Thus, while training on tne 
basic use of the console is not an Integral part of this train- 
ing course, it was considered desirable to Incorporate practice 
on the operations that are most frequently used to input Query 
Language . 

1.4 THE TRAINING STRATEGY 

Due to the dynamic structure and large size of the data 
base for System 473L, the dynamic nature of Query Language 
Itself, the frequent turnover of Air Staff personnel, and the 
vagaries of their normal and emergency duty schedules, effec- 
tive and efficient training of Air Staff personnel as profi- 
cient users of Query Language requires an on-the-job training 
capability. 



1.4.1 The Need for a Programed^ Self -Instructional Text , The 
need for maximum flexibility In the required training schedule 
to adapt to the needs of Individual trainees. In terms of both 
Individual learning rates and Individual schedules of Air Force 
duties, suggested the desirability of developing programed, self- 
Instructional materials that could be used for on-the-job training. 

1.4.2 The Need for an Operational Specification for Computer 
Directed Training , Since System 473L Is an Information processing 
system, it is capable of implementing both training and evaluation 
functions. The desirability of using the system itself as an 
adaptive training device, in conjunction with programed training 
materials, was suggested by several considerations: 

1) The design for the second-generation System 473L has certain 
physical characteristics that make it especially amenable to perform- 
ing training and evaluation functions. The computer's Integrated 
console contains a keyboard to permit operator inputs and an overlay 
board to permit access to the internal program. The overlay Is 
especially useful, since a special panel, can be cesigned to provide 
the trainee with direct access to and control over the computer 
directed training functions. The computer can be programmed to 
evaluate trainee inputs. In addition, the integrated console has a 
cathode ray tube output that permits alphanumeric displays. Such 
displays can be used to present both instructional (cue) and evalua- 
tive (test) materials. Further, the system permits time-sharing, 

so that more than one training station could be made available at 
any one time; 

2) Letting the trainee use the system itself for training 
would. It Is believed, make the training materials Inherently mere 
interesting and, as a result, increase the trainee's motivation to 
complete the course. It should also enhance the trainee's profi- 
ciency in performing specific system operations. Enhanced facility 
in operating the system would probably increase the frequency with 
which trainees would use the system in the future; 

3) The design for the System 473L computer permits more 
flexibility than conventional training materials in adapting the 
content and sequence of instruction to fit individual needs. 
Remedial sequences and content can be contingent on an evaluation 
of a series of criterion responses ; 

4) In addition, a computer can be more flexibly responsive 
than a text in complying with trainee requests for particular 
training materials. That is, a computer is more adaptable to 
learner-controlled instruction. This is a significant advantage 
not only for the training phase of a course, but more especially 



for proficiency maintenance training, when the trainee is more 
capable of self -evaluation and determination of necessary remedial 
materials; 

5) Not only can knowledge of results be Immediate, but 
analysis-feedback can be provided to the trainee based on his 
specific past performance; 

6) Due to the computer's planned capability for rapid data- 
analysis and its large storage capacity, detailed recording and 
analysis of trainee responses and, therefore, increased accuracy 

In assessing trainee proficiency and determining needs for specific 
revisions in the training materials. Is more feasible than with 
conventional materials; 

7) The potential high-speed printout capability provides auto- 
matic records of student progress; 

8) A further, potentially significant advantage of a computer 
directed training capability Is that, with some adaptations. It 
could also be used to provide training In other uses of System 
473L. 

Pretest data from the tryouts conducted with the programed 
text for OTC Query Language^ Indicated that the potential trainees 
are relatively homogeneous In terms of their prior knowledge of 
Query Language, e.g., on the pretest used for the initial tryout, 
no trainee scored higher than three per cent. Therefore, com- 
puter directed training was considered most desirable for the 
final phases of training, when there would be a greater need 
for Instructional flexibility due to Increased variability In 
trainee proficiency. However, It was felt that the Increased 
motivation for training and the projected Increase In future 
use of the system due to Increased operator proficiency would 
justify the additional expenditure necessary to provide computer 
direction for a ll phases of training, provided that this training 
capability would not impede the operational capabilities of 
System 473L. 

In view of the above advantages of a computer directed 
training capability, the Air Force decided that a detailed 
operational specification should be developed to describe such 
a capability and this operational specification should then be 
evaluated to assess the feasibility of Implementation in terms 
of the Impact on System 473L operational capabilities and the 
scope of programming required for this capability. Since the 
need for the programed, self -instructional text was Immediate 
and the feasibility of computer directed training was uncertain. 



^A description of the development and evaluation of the programed 
text for OTC Query Language Is presented in reference 1. The 
text Itself is presented In reference 2. 



the Air Force indicated that the programed instructional text 
materials should be developed first and that they should consti- 
tute a self-sufficient package, not dependent on any aspect of 
the computer directed training package. 



Section II 
THE DEVELOmENT OF THE PROGRAMED TEXT^ 

The development of the programed text involved several suc- 
cessive phases of work: the selection of course objectives and 
content; the selection of training strategies; the drafting of 
the programed materials^ and initial program tryouts and revi- 
sions; and the field tryout and revision. These phases are 
described below. In addition, this section describes the final 
program and supplementary materials, and evaluates the effec- 
tiveness of the text. 

2.1 THE SELECTION OF COURSE OBJECTIVES AND CONTENT 

The primary objective of the programed instructional text is 
to enable trainees to write statements in Intermediate Query 
Language, Model II, that could be used to retrieve information 
needed to solve moderately difficult resource management problems, 
stated in English. Subcriterlon objectives that were prerequisites 
to the primary objective included an orientation to the 473L System^ 
and knowledge of the organization of information in the data files 
in computer storage. 

Before preparing the draft instructional program^ it was 
necessary to define the specific content of the materials that 
would most effectively and efficiently reach the established 
terminal objectives. Then^ appropriate training strategies were 
selected on a topical basis, according to their appropriateness 
for each topic. Since Query Language transitioned from Model I 
to Model II during the period of development, the original outline 
and training strategies were specific to Query Language, Model I; 
as Query Language evolved, changes were made in the program to 
correspond to the changes in content. Thus, revisions during 
program development required not Just changes in training strategy 
but also substantial changes in program content. 

2.2 THE SELECTION OF TRAINING STRATEGIES 

Training strategies are based on the interaction of three 
factors: trainee characteristics, the course objectives, and 
specific characteristics of the course content . 

2.2.1 Trainee Characteristics . The following profile describes 
pertinent characteristics of the population of new trainees who 
will use this course: most trainees will have reached a minimum 
educational level of 12 years of public schooling; they will have 
received a 10-hour indoctrination course on the 473L System, 

^The programed text for Query Language, Model II, is presented 
in reference 3- 



Including a one-hour orientation lecture on the use of Query 
Language; they will have received no formal training In the use 
of Query Language; they will have already acquired limited pro- 
ficiency In using the Integrated console In the overlay mode of 
operation • 

Since Initial variability of potential trainees In terms of 
their prior knowledge of Query Language Is negligible^ It did not 
appear urgent for the content of Instruction to be varied for 
Individual trainees. That Is^ there seemed to be no need to use 
special techniques such as branching or gating to accommodate for 
wide differences In previous knowledge. 

In writing the Instructional materials^ an attempt was made 
to gear the reading difficulty to the ninth grade level In order 
to facilitate training efficiency. This was done by exerting 
control over sentence length; checking the level of Infrequently 
used words and, whenever possible, replacing those whose level 
was unusually high; and, on a more empirical basis, by revising 
those steps which appeared difficult In terms of Initial tryout 
data.-* 

2.2.2 The Course Objectives . The primary objective of this 
course, as mentioned previously. Is to develop proficiency in 
writing Query Language statements for a representative set of 
problem types. This required the development of verbal, con- 
ceptual skills. The instructional programing techniques selectee 
as most appropriate were the use of small steps, careful se- 
quencing, overt written responding (with responses building in 
complexity as more material is learned), immediate confirmation, 
and self -pacing. The selection of this methodology was based on 
previous experience at the Institutes with its application to 
similar training problems. 

2.2.3 Characteristics of the Course Content , The selection of 
an overall programing methodology was primarily dependent on 
trainee characteristics and the terminal objectives. More spe- 
cific training strategies were selected on the basis of the 
characteristics of the subject matter. 

One of the strategies contingent on the course content was 
the extent to which the data in computer storage should be sampled 
by the training problems. The stored information in System 473L 
Is extensive and AF documents are available to aid the operator 
in determining the location and format of the data to be accessed. 
Further, some of the data files have varying frequency of use; an 
individual file might be used rarely by some Air Force personnel 
and very frequently by others, depending on the primary function 

^he grade level of words was determined by referencing Thorndlke, 
E. L. & Lorge, I. The teacher's word book of 30,000 wo rds . 
New York: Columbia University, 1959^ 
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of the particular user group (e.g.^ logistics, personnel, etc.). 
Therefore, it was decided that while basic training should exhaust 
the possible capabilities and variations in format of Intermediate 
Query Language, this training could not feasibly be applied to an 
exhaustive sample of the actual data in computer storage. There- 
fore, the range of data for which the trainees were to write 
Query Language statements during the training course was not 
intended to be exhaustive but to constitute a representative 
sample of the types of information in computer storage. This 
approach was expected to minimize training time by reducing the 
redundancy inherent in an approach that sampled all data. To 
provide for effective transfer of training for the retrieval of 
types of data not specifically covered in the training course, 
the trainees were to receive extensive practice in the use of 
the AF documents specifying the location and format of data. 

The sequencing of training steps was dependent on the 
intrinsic interaction among the items to be covered. For example, 
understanding the structure of a Query Language statement requires 
knowledge of the organization of information in computer storage. 
Therefore, the structure of the computer files was taught early in 
the course. As in this case, whenever possible sequencing was 
designed to optimize the amount of positive transfer from one con- 
cept to another. 

2.3 DESCRIPTION OF THE FINAL PROGRAM 

2.3.1 T he Preprogr am . A preprogram of 3^ frames was oeveloped to 
familiarize the trainees with the characteristics of the textual 
program and take the place of an initial orientation lecture on 
self -instructional programing. This preprogram is designed to be 
taken prior to the programed books on Query Language and would 
enable trainees to start the Query Language textual program with- 
out any additional assistance. Thus, the preprogram reduces to a 
bare minimum the need to utilize the time of training personnel. 

2.3.2 T he Textual Program on Query Language . 

2.3.2.1 Number of frames . Appendix A shows the number of frames 
in each volume of the programed text on Query Language. The total 
number of frames, counting the preprogram, is l448. 

2.3.2.2 Content of frames . The program is cumulative in that the 
ability to write Query Language statements depends on a number of 
subcriterion skills, and, therefore, the sub criterion skills were 
developed first . 

The dependence of the criterion upon subcriterion skills is 
reflected by the sequence of topics in the program. Thus^ the 
initial sections of the program develop the subcriterion skills: 
successively, 133 frames are devoted to an orientation to the 
^73L System; 43 frames cover the organization of information in 
the data files; 21 frames cover the selection of the file from 
which the desired data might be obtained; and 51 franies cover the 



selection of the attributes and values that need to be specified; 
the major portion of the program, consisting of 977 frames, pro- 
vides intensive development of the ability to write Query Language 
statements. The last three volumes, consisting of I89 frames, 
give extensive guidance and practice in the use of the Air Force 
Data Control Manuals, so that the trainee can learn to write 
"complex" Query Language statements, which require great facility 
in working with a number of data files, and so that the trainee 
can become proficient in using these manuals and be able, in the 
future, to readily access data from files that he has not pre- 
viously worked with or from files that he is not extensively 
familiar with. 

The strategy used in developing proficiency in writing Query 
Language statements is to require that trainees learn the func- 
tional correspondence between the rules of format and the appro- 
priate computer operations. An Important factor in this develop- 
ment of proficiency is the ability to understand the organization 
of the data files. The importance of this factor is illustrated 
by the frames below, excerpted from Volumes XI and XII. For 
illustrative purposes, the frame answers are also shown here, 
enclosed in [ ] and/or underlined where appropriate. 

Volume XI: MIN and MAX Functions 

MIN and MAX as Values 

1. Another function which Query Language can perform is to select 
entries with maximum and minimum values of a particular attribute. 
To find an airfield with the longest runway, you would want to 
select the entry with the (minimum/maxim.um) [" maximum ] value for 
RNWY LENGTH. L J 



3. Which of the two entries below will be selected? 
(Airfield name) fHobsonl 

. . . WITH RNWY LENGTH = MAX . . . 

AFLD NAME HOBSON CHECKER 
RNWY LENGTH 15J2^00 11500 
RNWY WIDTH 150 200 

5. When MAX or MIN are used, the entry must meet all ot her 
qu alifications first . In the following example, Hobson has the 
Tongest runway. However, it does not qualify because it is not 
in the [ "united States] . The entry that will be selected is ("PowT 

... WITH COUNTRY = USA, RNWY LENGTH = MAX ... 

AFLD NAME HOBSON DOBSON HOWE DOW 
COUNTRY CANADA USA FRANCE USA 
RNWY LENGTH 13700 9500 9000 10000 

10 



l4 . If there is a tie for the minimum or maximum value, you can 
select between those entries by specifying a further minimum or 
maximum modifier. A second modifier will be in effect only if 
there is a tie resulting from the first modifier. In the fol- 
lowing example, circle the entries which will be selectee by the 
first modifier. Of the circled entries, which one will be selected 
by the second modifier? Place a check above that entry. 



WITH RNWY LENGTH = mx, RN WY i WI DTH = MIN 



AFLD NAKE 
RNWY LENGTH 
RNWY WIDTH 



ABLE 
l(2f(2f(2f(2f 
150 



BAKER 

12(2f(2f0 

20(2f 



(charlieX 
125(^0 1 
150 ) 



DOBBS 

10000 

250 



HIBBLER 
12500 
200 



MIN and MAX as Computed Attributes 
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WIT 




131FIS 


UNIT 


LOG 


AMARILLO 


MDS 




P100P 


ACFT 


RDY 


19 


CRWS 


RDY 


16 



Suppose you want to know whether the 131st FIS at A^:ARILLO 
has enough crews to man the operationally reacy aircraft. You 
can answer this by determining which of two attributes has a 
minimum value; the two attributes to be compared are 
TaCFT RDY] and [CRWS roYl . 

either order 
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UNIT 




131FIS 


UNIT 
MDS 


LOG 


AMARILLO 
F100F 


AGFT 


RDY 


19 


GRWS 


RDY 


16 



The values of ACFT RDY and CRV/S RDY may be compared within 
each column, or entry. Therefore, the minimum (or maximum) of 
these values may be determined for each column, or fe^nt^ryl . 



24. For a given set of alternate attributes, to specify the one 
with the maximum value, you would use [MAX] (attribute, attribute, 
to specify the one with the, minimum value, you would use PMIN 



(attribute, attribute. 



4 



■); 



27. For all tanker aircraft, we want to know the MDS, anc" the AVG 
TKOFF WT or the AVG LAND WT, whichever is lower. Gomplete the Query 
RETRIEVE AGFT GHAR WITH AGFT GAT = TANKER THEN LIST MDS, 
TmIN ( AVG T K OFF WT , AV G LAND WT )" 



]1 
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Volume XII: GREATEST and LEAST 

2. When an attribute may have more than one value per entry, and 
the values are numeric, we may not want to print out all the 
values. We may want to print out only the value which is greatest 
or, on the opposite extreme, the fleast (smallest, etc . )"] . 

5. When an attribute has more than one numeric value per entry, 
you can write a QL statement to find the fgreatest or fieastl 
value for that entry. L J L J 

either "order 

8. In the table shown below, for which attribute could you find 
the greatest or least value? fSTAGE CRWS RQ ^ 





PLAN 


PLAN IDENT 


1234a 


BASIC PLAN 


123 


MDS 


F10C^A 
F10j2fB 


STAGE CRWS RQ 


20 




15 



L 

IDENT file 


J 




1234B 


1235A 


1235B 


123 


123 


123 


F100A 


B32 


B47E 


F100C 


F100D 


B5S 


31 


16 


21 


10 


14 


25 



33. While attribute = MAX is designed to select the one entry 
that has the maximum value for the specified attribute, attribute = 
GREATEST selects one value of the specified attribute for each 
[ entry ], and a correspondingT value ] for every other attribute that 



belongs to the same [ subset! 



37. We want to know the plan identifications and all the sus- 
taining phase data available pertaining to the sustaining MDS's 
with the largest number of total flying hours for each entry in 
the PLAN IDENT file. Write the Query. Obtain a display in any 
format . 

RETRIEVE PLAN IDENT WITH TOTAL FLY HRS ^ GREATEST THEN 
L IST PLAN IDEN T, SUSTAIN MDS, TOTAL FLY HRS, TOT ELAPSED 
TIME n 

2.3.2.3 Reference materials . Aside from the illustrations which 
are presented on individual frames and the AF documents specifying 
the location and format of data, the textual program utilizes 36 
exhibits external to the programed books. Of these, 3b are bound 
together in one volume and are used to display diagrams, models, 
descriptions of data files, etc., that are frequently referenced 
by the program to avoid unnecessary repetition of material on 
individual frames. An additional exhibit that is bound separately 
consists of a series of review panels. The major types of exhibits 
and their use are described below. 
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a ) Schematic Dia gram of Information Flow In the 473L System . 
This diagram, shown In Figure 1, was prepared as an aid In devel- 
oping an understanding of the 473L System, In which Query Language 
Is a major mode of data retrieval. This diagram shows the possible 
flows of information in the 473L System when Query Language Is used 
for data retrieval: It shows the types of input components, the 
flow of input through the computer to the storage files, and the 
flow of output data to the output component. This aid is used 
exclusively in the first part of the program, which presents a 
brief orientation to the 473L System. 

b) Illustrations of Various Sys t em Components . These exhibits 
are Intended to facilitate the 'trainee 's orientation to the 473L 
System. 

c ) A Descriptio n o f the Procedural Steps Required to rle_tri_eve 
D ata for a Specifie d Problem Us ing Query _Language . This exhibit , 
shown in Figure 2, gives a brief, step-by-step description of the 
overall process of problem solution, including an Initial problem, 
the operator's actions, the operation of the computer, and the 
output of data. 

c) A D escription of th e Data Files. This exhibit is usee to 
help orient the trainee to the names of the data files anc their 
general contents. An excerpt from this exhibit is given in 
Figure 3» 

e) Charts Showin g the Con ceptu al Categ oriza tion of the Data 
F iles . These charts group the data files according to subject 
matter (e.g., airfield information, personnel information, etc.) 
and according to the level of data description (e.g., narrative 
descriptions, detailed characteristics, index data, etc . ) . These 
are used to help the trainee become proficient in accessing the 
proper file . 

f ) Sample Tables with an Incom pl ete Lis tin g of A t tributes 
a nd Typ ica l Values . These tables are used in the program to 
develop the trainee's concept of the organization of data stored 
in the data files. One such table is shown in Figure 4. Shorter 
excerpts from tables are frequently shown on individual frames. 

g ) An Example of How Informa ti o n_ is Re trieved fr om the Data 
Files Using Que ry La n guage . This exhibit, shown in Figure 5> 
illustrates "the overall retrieval process by presenting: a sarr:ple 
set of files, including an excerpt (with hypothetical values) from 
the file from which information is to be retrieved for the solu- 
tion of a problem; and, the identification of each element of a 
Query Language statement with the successive selection from these 
computer storage files of a particular file, particular columns 

of data, and the desired output data. 

h) The Basic Output D irec tors , This exhibit shows the basic 
output directors, the output device for each, and the format cf 
output data. 
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i) Examples of Corrective Action Following Error Message s, As 
part of the initial orientation to the 473L System, an exhibit Is 
used to illustrate the type of display on the integrated console's 
CRT display screen for several different types of error, and the 
action that should be taken in each case to correct the erroneous 
Query Language statement. 

J ) Rules, Formats, Examples of Input and Output, and Other 
Te aching Aids for Special Query Language Functions (such as GCD , 
SUM, titles, sorts, special directors using the SAVE table, CHECK, 
and Complex Queries). These exhibits are used to help the trainee 
learn and discriminate between the different formats and functions 
of each Query Language function. Examples of these exhibits are 
Figures 6 and 7. 

k) Review Panels . These are Intended to be used for a cumula- 
tive review of the first JQ^. of the program (through Volume XIX), 
They summarize the information taught about the sequencing and 
functions of the basic Query Language elements, such as the file 
indicator, the qualifier, etc.; the organization of data In the 
data files; the output devices and formats; and, the function, 
format, rules for use, and relation to the other parts of a state- 
ment, of specific parts or types of the basic elements, such as 
attributes, special kinds of attributes (e.g., SW. and GCD), values, 
etc . 




QL ENTRIES 



PUNCHED 
CARDS 



QL ENTRIES 



HIGH-SPEED 
PRINTER 



QL ENTRIES 



L-3055 
COMPUTER 

lAN/FYQ-ll) 



QL ENTRIES 



I/O 

TYPEWRITER 



QL ENTRIES 



CRT DISPLAY 
SCREEN 



MULTICOLOR 
DISPLAY 
SCREEN 



INTEGRATED CONSOLE 

(IC) (AN/FYA-2) 



CONTROL KEYS AND LIGHTS 



ALPHANUMERIC 
(A/Nj KEYBOARD 



PROCESS 
STEP KEYS 



W 



OTHER 
CONSOLES 



CONSOLE 
PRINTER 



Pig. 1. Schematic diagram of planned information flow in 
the design for a second-generation 473L System. 
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step 1. Vie define the problem. In this example, we need to fjnd 
out the names of all airfields with runways of at least 10,000 ft 
This information is stored somewhere in the 3055 data files. All 



we need to do is get it out. 



STORAGE UNIT 



L-3055 

DATA 

BASE 




Millions of items 
of information -- 
>- similar to a large 
library -- revised 
every day. 



L-3055 COMPUTER 



Step 2 



The operator selects a program (PIETRIEVE) and the correct 
file (airfields) from a description of the data baije in specially 
prepared AF documents • 




Step 3^ The operator selects the correct attribute names (RKWY 
LENGTH and AFLD NAKE) fi-^om a description of the data base. 




> - 



Step U» The operator writes a Query Language statement. 

RETRIEVE AIRFIELDS WITH RNWY LENGTH > 10000 
THEN LISTV AFLD NAME, RNWY LENGTH") 

Step 5. The operator types In the QL entry on the Integrated Console 



TO COMPUTER 5 




Fig^ 2. A description of the proposed procedural steps required 
to retrieve data for a specified problem using Query Language. 
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step 6. The computer selects the RETRIEVE program and starts 
processing the QL statement. 



I r 1 

PROGRAMS ! 
RETRIEVE ^^ 



! PROCESSING 



UNIT 



^^^ QL STATEMENT 
^ FROM IC 



L-3055 COMPUTER 



Step 7 



The RETRIEVE 
pr 0(2; ram 
selects the 
correct 
file. 



AIRFIELDS 



PROCESSING 



DiTA_ j> UNIT 

RETRIEVE 
PROGRAM I 



L-3055 COMPUTER 



and the 
correct 
attribute 



and the 
correct 
qualifying 
columns , 

and the 
correct 
output 
attributes 
specified by 
the selector. 



I 



AFLD NAME 



AIRFIELDS 



RNWY LENGTH 



LENGTH ^^'°°° ^^'°°° ^'^^^ ^'°°° 



:b 



>|afld name| (andr~ws) 




] 



step 8. The computer sends the output to the selected devJ.'ce 
( LISTV specifies the CRT display screen of the IC). 




L-3G55 COMPUTER 



INTEGRATED CONSOLE 



Step Q. The desired data is outputtea in the desired fonnat 
(LISTV specifies vertical format). 




CRT DISPLAY SCREEN 

Pig. 2 (cent. ) 
l6 



File Name 
Abbreviation 



File Name 
Clear 



ACFT CHAR 



ACFT MSL MAINT 



ACFT PLAN FAX 



AFLD CLIM INDEX 



AFLD CLIM REEL 



AIRFIELD INDEX 



AIRFIELDS 

AIRFIELD CLIM 

AIRFIELD REEL 
AIRMEN AFSC 



Aircraft 
Characteristics 



Aircraft/Missile 
Maintenance 



Aircraft 
Planning 
Factors 



Airfield 
Climatology 
Reel Index File 



Airfield 
Climatology 
Reel File 



Airfield Tape 
Reel Index File 



Airfield Disk 
File 



Airfield 

Climatology 

File 

Airfield Reel 
File 



Descript ion 



Physical, operational, and 
logistic information on USAF 
and civilian reserve aircraft . 

Schedules for normal and coni- 
pressed rates of maintenance 
in a depot for each MDS, and 
the origin and destination 
unit of the aircraft. 

Factors used in solving opera- 
tional problems dealing with 
aircraft/airfield compatibility 
and traffic compatibility, and 
the planning for contingencies 
that involve aircraft anc war 
materiel items . 



Location o^ current 
tape reels, and the 
in which data for a 
country is located . 



and backup 
reel number 
given 



Climatic data such as tempera- 
ture, precipitation, and fre- 
quency of occurrence of speci- 
fic ceiling and visibility 
conditions by month for approxi- 
mately 3500 airfields. 

Location of current and backup tape 
reels, and the reel number in 
which data for a given country 
is located . 

Basic , frequently used cata 
concerning airfields (rnwy 
data only for longest runway). 

Monthly meteorological data 
on airfields in the disk and 
reel airfield files . 

All data concerning airfields, 
including climatology data. 



Airmen AFSC (Air A breakdown of the AFSC's, 

Force Specialty with the meanings of each 

Code) File part of the code. 

Fig. 3. A partial listing and description of the data files. 
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AIRFIELD REEL* 



AFLD NAME 




ANDREWS 


LOGAN 


MACDILT, 


COUNTRY 




USA 


USA 


USA 


STATE 

AVGAS CAP** 
RNWY LENGTH 


1 
2 
3 


MD 
200 
12000 
11^ 


MASS 
150 

10^ 


FLA 

180 

9^0 

8050 

7500 


RNWY WIDTH 
RNWY SFC 


4 
1 
2 

3 
4 

1 


600(2f 

230 

200 
190 
150 
CONCRT 


210 

Hi 
180 

CONCRT 


190 
210 
180 

ASPHALT 




2 


CONCRT 


CONCRT 


CONCRT 




3 


ASPHALT 


CONCRT 


ASPHALT 




4 


ASPHALT 






NAVAID 




TACAN 


TACAN 


ILS 


FACILITY 




AIR/GND 
TOWER 


GCA 
TOWER 


TOWER 



*One cJifference between the AIRFIELD REEL file ancJ the AIRFIELDS 
file is that the AIRFIELDS file is on cJisk and contains runway 
cJata for only the longest runway^ but the AIRFIELD REEL file is 
on tape ancJ contains data for all runways. 

**Values for AVGAS CAP are in thousands of gallons (K GAL). 
Fig. 4. An incomplete table with typical values. 
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RETRIEVE 



AIRFIELDS 



WITH 



/ L.3955 


/ PROGRAM NAME 


RETRIEVE 


PDR 


RCDR 


(':.' il<.l!.< 


.,, vIIm ,.i 


fH, M, 


m:.,.,i,., ^11,. 


, 


■■'1 ■■. ) 


(II.I.O (.Ilnl 


.,mI,m y! 


(N.lto vV 


Molln nlluilh 


n<ii<iiii) 


IK. Ill, nil. 


(>.... II,, ..ll,M.I 


t'.M 


.11.' <).... 11. 


Mh,)M i!..I (II 


. [v.i.;i 


II.. ..1 


l>>(> Ilo nil 


■■• <•! 


..t iHcyiM 


ylUt oltnii 


(tllolllt 


)mIm> ylN. 



\ DATA BASE 


V FILE NAMES 




ACFT CAT 

v::.. .'1 


AIRFIELDS 


PLAN IDENT 

••t:..!'., (> 1 

ili.> ylli. nl.i.M 


FORCE 
STATUS 

(11.1 ii..n.. ()w 

..II.. ..)..> )ii.. ..1.0 

(ll.lin <.1|.)l Im<V 






I)!).) ollu!!.) 


iir.ni.JMt ylU. 


r.lclhho ollolln 


iM.llo iillniiho y 






(II,! II. tl. 


. ..tl.M.I... )!' 


<}...'ll.. ..II....I... 


.Iln 1.II....I... vllx i- 






..II, .,.!.,) ^11.. 


iioi <:■.:... .. 


..1 ..1 (IM.. 


.. II.. 1 <M.I \\i.\' 






<I1.lii> (.ilul 


.'!.. nil. .nil.. 


I>><) 11. > mIIi.i 


nnlld Mlli.iiitn y! 






ni.!'in iiJlnn 


oi).).. rule; 


Ini yi1(. oi.nii I! 


i1(. olinii ilolcliti 













DATA IN 
AIRFIELDS 
FILE p 



COUNTRY=USA 



ELEV<iqq 



-4----\~--}- — \ 



COUriTRY 

AFLD NW-IE AIJDRKVvT 

STATt: IID 

ELEV 13^ 



I USA I I UGA I I USA I 1 U3A | 



LOGAN I ! SHAW I ALTUG 

! ' 

MAGS ' I GA I OK LA 

< — — pH 



ca:jada ca:jada 



HILL 



I ^^ I I 90 I i^^30 75 



DOVER 



28(3 



THEN LIST 



AFLD NAME 



QUALIFIED AND 
SELECTED DATA 



OUTPUT : 



COU.JTRY UGA UGA 

AFLD NAM;; -^— LOG>VW-#— SHAW- 
3TATE '1AG3 "^tA 



ELEV 



ELEV 



ss ■ 



9v) -^ 



KAFLD ;IA;'^; LOZAH GiiAW A 



Fig, 5. An example of how Information is to be retrieved 
from the data files using Query Language. 
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format 



functions 



a. simple form — one SUM control 
attribute and one summed 
attribute, 

. . . SUM BY SUM control attribute 

(summed attribute) ... 

Example ... SUM BY UNIT (ACFT POSS)., 

b. general form -- multiple SUM control 
attributes and summed attributes. 

. . . SUM BY SUM control attribute EY 
SUM control attribute BY SUM control 
attribute (summed attribute, summed 
attribute, . . . ) ... 

Example . . . SUM BY UNIT BY MDS (ACFT 
RDY, CRWS RDY) ... 

c. short form (no SUM function in the 
qualifier) . 

... SUM (summed attribute, summed 
attribute, . . . ) ... 
Example ... SUM (ACFT RDY, ACFT 
POSS ) . . . 



Will sum values of 
summed attribute for 
each value of the SUM 
control attribute. 



Will sum values of each 
summed attribute for 
eveiy combination of 
values of the SUM con- 
trol attributes. 



Will provide a total sum 
(over all entries that 
qualify). 



d. short form if a SUM function has 
been specified in the qualifier. 
. . . SUM . . . 

Example ... LISTH UNIT, UNIT LOG, 
SUM] 



Outputs sums already 
computed in the qualifier 



Fig. 6. Formats and functions of SUM in the output selector. 
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FOB (C) 
ITEM (C) 



STORAGE LOG 



SL ASSET 



(S) 



entry 

PARKERSBURG 
CHAFF RRI31 



ARNOLD MULDOON VICTORY 



5000 7000 



3000 



entry 

SAMPSON 
CHAFF RRI31 



ARNOLD VICTORY 



4000 



7000 



1. RETRIEVE MATERIEL STATUS WITH ITEM = CHAFF RRI3I, STORAGE 
LOC = VICTORY THEN LIST SL ASSET "I 

output : 



SL ASSET 



2. RETRIEVE MATERIEL STATUS WITH ITEM = CHAFF RRI3I, STORAGE 
LOC = VICTORY THEN LIST SUM (SL ASSET)' 



output 



SUM SL ASSET 



3. RETRIEVE MATERIEL STATUS WITH ITEM = CHAFF RR131 THEN LIST 
SUM BY STORAGE LOC (SL ASSET)"! 



output 



STORAGE LOC 
SUM SL ASSET 



Fig. 7. Examples of Input and output formats for SUM. 
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2.3«2.U Program format . Each "frame" consists of two parts, a 
stimulus panel and an answer panel. A stimulus panel consists of 
written material, including at least one incomplete statement; 
missing words or phrases are indicated by blanks. Answer panels 
contain the missing words or phrases. In some cases, where this 
appeared appropriate, the missing words or phrases are presented 
in their appropriate context (from the stimulus panel) on the 
answer panel. 

Stimulus panels, and likewise response panels, are presented 
three to a page, on sequential pages of bound volumes. Frames 
that relate to a relatively discrete topic are bound together; 
thus, separate topics are usually denoted by separate volumes. 
The sequence of stimulus and answer panels is shown in the schematic 
on the following page. 

2.3.3 Computer Exercises . Although developing proficiency in 
using the integrated console is not a primary task of the pro- 
gramed text, it was suggested that some experience in actually 
using the console to retrieve data with Query Language statements 
would have a motivating affect. In addition, the exercises are to be 
used to permit trainees to become adept at interpreting and 
reacting to Error Messages related to Query Language inputs, dis- 
played on the CRT display screen. 

The computer exercises are designated for use at points 
scattered throughout the course. The instructions for these 
exercises are bound together in an Exercise Book. 

2.3.4 Test . Two tests were developed for use in evaluating the 
Query Language course. One was used during the preliminary try- 
outs to test the first 70^ of the program. The other test is the 
final posttest, which is cumulative over the entire content of 
the program; a copy of this test is shown in Appendix B. A copy 
of the final test answer and scoring key is in Appendix C. 

The final posttest is designed primarily to evaluate pro- 
ficiency in writing Query Language statements; it requires the 
trainee to write a representative sample of the types of Query 
Language statements taught throughout the program. This test 
was used at the completion of the programed course to evaluate 
the overall effectiveness of the course, and to indicate neces- 
sary specific revisions in the program. In the future, this 
test can be used to evaluate the proficiency of individual 
trainees. The almost exclusive emphasis in the final test on 
proficiency in writing Query Language statements reflects both 
the importance of this objective and the fact that the level of 
proficiency in writing Query Language statements also reflects, 
in most cases, the level of proficiency in the subcriterion 
behaviors . 
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2.4 THE DRAFTING OF FRAMES, INITIAL PROGRAM TRYOUTS, TECHNICAL 
REVIEW, AND REVISIONS 

Draft frames were based on the initial specification of 
course objectives, content, and training strategies. As succes- 
sive portions of the program were completed, they were given pre- 
liminary tryouts and a review for technical accuracy, and revised 
on the basis of the tryout data and technical comments. Due to 
the evolving nature of Query Language, some changes in frames 
were also necessitated by changes in the language itself. 

Prior to the field tryout on the entire text, there were two 
preliminary tryouts and revisions. The volumes evaluated in the 
first tryout constituted about 38j^ of the final text. The volumes 
in the second tryout included the volumes used in the first tryout 
plus additional materials; the total number of volumes in the 
second tryout amounted to about 70jt of the final text. The two 
preliminary tryouts and revisions are described below. 

2.4.1 Subjects . All of the preliminary tryouts were conducted 
at the Pentagon with a small number of Air Force personnel who 
were typical of the target population. Four trainees participated 
in the first tryout. In the second tryout, these trainees took 
only the materials that had been drafted subsequent to the first 
tryout; two new trainees took all of the materials that had been 
drafted up to this point (approximately lOfo of the final program). 

2.4.2 Results . Since the number of subjects for the preliminary 
tryouts was small, the results were used for the very limited 
purpose of revising the programed text. Generalizations to other 
programed materials are not appropriate. 

Percent correct pretest scores for individual trainees were 
low: the median score for the six trainees participating in 
the two preliminary tryouts was 2.99^; the range was 0.5/^ to 10. OJ^. 

For the preliminary tryouts, the average error rate on each 
draft volume of the programed text (not counting Volume I - the 
preprogram, for which the error rate was negligible) is given in 
Table 1 below. The error rates ranged from 3.65^ to 24.2^. The 
median error rate for the draft volumes was 8.5?^. As shown, 
only two out of l4 volumes had an error rate exceeding 10^.. 

Volumes II through IX were used in draft form in the first 
tryout; the average error rate on this portion of the program 
was 8.35^. In addition to the revised volumes II-IX, Volumes X 
through Xrv were used in draft version in the second tryout; 
the average error rate on draft volumes X-XIV was 7.6^. 
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TABLE 1 

Average Error Rates on Draft QL Volumes That 
Were Used in the Preliminary Tryouts 

First Tryout Second Tryout 

Rate Volume Average Error Rate 

X 8.8^ 

XI 8.1 

XII 4.7 

XIII 7.0 

XIV 8.7 



Volume 


Average Err 


II 


6.6^ 


III 


8.5 


IV 


23.4 


V 


9.1 


VI 


9.6 


VII 


3.65 


VIII 


24.2 


IX 


5.2 



Individual error scores on the first 70^? of the draft program 
and percent correct posttest scores on the test covering this por- 
tion of the program are given in Table 2 below for each preliminary 
tryout . 

TABLE 2 

Percent Error for Each Trainee on the Program and Percent 
Correct on the Test for Each of the Preliminary Tryouts 



Tryout 1 


Si 


S2 


S3 


S4 


Draft Program 


9.9^ 


6.0^^ 


9.9^ 


7.5^ 


Test 


96^ 


95r 


9C¥ 


19^ 


Tryout 2 


^5 


S6 






Draft Program 


2,&fo 


6.2^ 






Test 


92^ 


88^ 







The average posttest score for the first tryout was 90% and 
likewise for the second tryout. In view of the high posttest 
scores and the fairly low error rates on even the draft volumes -- with 
a couple of exceptions -- and considering the apparent lack of 
relationship between program error rates and proficiency on the 
posttest, revisions were mostly based on the technical review and the 
critical comments of the trainees . 
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2.5 FIELD TRYOUT AND REVISION 

2.5.1 Design . While the scale of the field test was restricted 
by the limited availability of personnel representative of the 
target population^ the scale was sufficient since the field test 
was used exclusively for very specific evaluative and diagnostic 
purposes: it was used to evaluate the effectiveness of the pro- 
gramed course in terms of the specified objectives, and to diag- 
nose areas of Query Language that trainees had difficulty in 
learning, for which revisions were required in the training mate- 
rials. 

As previously mentioned, the primary objective of the program 
is to enable Air Staff personnel to write Query Language statements 
for the solution of moderately difficult, specified problems. 
Evaluation of the program's effectiveness in achieving this objec- 
tive was based exclusively on the trainees' proficiency on the 
criterion test, which emphasized ability to write Query Language 
statements. 

2.5.2 Subjects . As shown in Table 3 below, seven of the 
eight trainees had taken the earlier programed course on OTC 
Query Language, which was the first operational version of Query 
Language; five had had prior training on either Model I or 
Model II Query Language; and, three of the trainees had 
actually participated in the preliminary tryouts with the draft 
materials covering approximately 70^ of the program on Inter- 
mediate Query Language, Model II. 

TABLE 3 

Background Data for Trainees Participating in the Field Tryout 

Participated in the 
Preliminary Tryouts on 
the Draft Program for 
Model II QL 

Yes 
Yes 
Yes 

No 
No 
No 
No 
No 



Since the structure of Query Language changed greatly from 
OTC to Model II, the previous training of some subjects in OTC 
Query was not seen as a serious problem; in fact, in transitioning 
from OTC Query Language to Query Language, Model II, it is 
problematic whether transfer of training was positive or negative. 
In either event, there was an overriding, pragmatic Justification 

26 





Previous 


Previo 


us Training 




Training 


in 


either Model I 


Trainee 


in OTC QL 


or 


Mod 


el II QL 


Si 


Yes 






Yes 


S2 


Yes 






Yes 


u 


Yes 






Yes 


Yes 






Yes 


sz 

St 


. Yes 






Yes 


Yes 






No 


S7 


Yes 






No 


4 


No 






No 



for the selection of these particular trainees to participate In 
the field tryout: the tryout was utilized by the Air Force as 
an opportunity to meet current training needs. 

All of the trainees completed the programed course on 
Intermediate Query Language, Model II, but only five took 
the final test. These were the trainees Identified In Table 3 
as 82^ So, Sas Srjy and Sg. 

2.5^3 Administration of the Field Tryout . The tryout was admln- 
Istered by Air Force personnel. Aside from an Initial briefing, 
direct monitoring was negligible. Work on the course was done at 
the Pentagon, on a flexible schedule. Other Air Force duties 
were Interpolated, as required, by each Individual . Interspersed 
activities consisted of the trainees' normal duties at the 
Pentagon In their regular section. Trainees worked on the course 
an average of 6-1/2 hours per day. Total training time varied 
from a minimum of five days to a maximum of eight days. Since 
there was no significant relationship between program error rates 
and test proficiency in the preliminary tryouts of the program, 
and since the collection and analysis of program error data is 
very time-consuming, it was not considered worthwhile (and it 
was definitely not expedient) to collect program error data. 
The only data collected from the field ti^out were program com- 
pletion times and percent correct scores on the final test. 

After the instructional program was completed, each trainee 
was given a posttest designed to evaluate his proficiency in 
writing Query Language statements. As mentioned previously, a 
copy of this test appears in Appendix B. 

2.5.4 Results of the Field Tryout . 

2.5.4.1 Completion times for the programed materials . Since 
interruption of piogress on the program to perfo-'^rr other duties 
was intermittent, recording exact times would have been burden- 
some. Therefore, the average number of hours per day was esti- 
mated, at 6-1/2 hours, and the completion time for each trainee 
was recorded in terms of the total number of days required to 
finish the program. The completion time for each individual 
trainee is shown in Table 4 below. The average number of days 
for completion was 6.3 days; at 6-1/2 hours per day, this meant 
an average of roughly 41 hours to complete the program. 

TABLE 4 

Program Completion Time for Each Trainee 

Trainee Time Trainee Time 

Si 6-1/2 days S5 7 days 

S2 6 days Sg 5 days 

So 7 days Sy 6 days 

Si4 5 days Sq 8 days 
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2.5.^.2 Scores on the final posttest . As mentioned previously, 
the final test was taken by only five of the trainees partici- 
pating In the field tryout. Scores for Individual trainees on the 
final test are shown In Table 5* 

TABLE 5 

Percent Correct Scores on Final Test for Individual 
Trainees Participating In the Field Tryout 

S2 sh.e$> 

So 46.5 

si 80.2 

Sy 86.8 

Sq 72.4 

The average percent correct score on the final test was 76. IJ^. 
The median was 80^, with scores ranging from 46.55^ to 94. 65^. Thus, 
scores on the field tryout final test were lower than those obtained 
on the posttest covering 'JQ^ of the program, which was used In the 
preliminary tryouts. The most probable explanation for this Is 
twofold: 1) the last 30^ of the program had not previously been 
tried out; and, 2) the last 30^^ of the program was Intrinsically 
more difficult, overall, than the first 70^^ and the final test 
was correspondingly more difficult than the posttest used In the 
preliminary tryouts^. 

2.5.5 Revisions After the Field Tryout . Since time was a critical 
factor In completing the revisions, revisions were restricted to 
those based on data from the final test. 

The first step In revising the program was to make a few 
additional technical revisions based on Information obtained from 
Air Force experts on Query Language. Further revisions In the 
program were based on errors made on the final test. For each 
Item missed on the test, corresponding topical sequences were 
Identified. Appropriate revisions were made In these sequences. 
The revision based on test errors was considered of critical 
Importance and ample time was devoted to this work In order to 
adequately revise for all recurrent test errors. 

5as seen In Table 3^ trainees S2 and S^ had previously participated 
In the preliminary tryouts of the draft program; their scores on 
the test covering the first 705^ of the entire program were 90JI^ 
and lSff(>9 respectively. The substantial drop In S3's score would 
seem to Indicate that the final test was substantially more 
difficult (for him) than the posttest used In the preliminary 
tryouts . 
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2.6 DISCUSSION AND RECOMMENDATIONS 

Generally, the results of the field tryout indicated that the 
programed course can train to a satisfactory level of proficiency 
over a period of about six days, using a massed training schedule 
of roughly 6-1/2 hours per day. Past experience with a similar 
programed course on OTC Query Language indicated that a training 
schedule requiring less time per day achieves a slightly higher 
level of proficiency^. Thus, if the immediate need for personnel 
trained in Query Language is not especially urgent, a spaced 
schedule of about two to four hours per day would be most desir- 
able. But if immediate training is critical, a massed schedule 
could be used with only a small penalty expected in terms of the 
final proficiency level. 

For the implementation of this course as a training device, 
it is recommended that the final test be retained for evaluative 
and motivational purposes. It is also recommended that the inte- 
grated console exercises be used to demonstrate data retrieval 
for a variety of Queries. While these demonstration exercises 
Would be on the computer, they would not be computer directed. 
If it is possible to provide the adaptive, computer directed 
training course described in the next section, this would be 
recommended in preference to the programed text. 



^A massed practice group (approximately seven hours per day) scored 
84^ on the posttest and had an average error rate of J .0^ . A 
spaced practice group (approximately two hours per day) scored 
90^ on the posttest and had an average error rate of 5*Of* More 
detailed data are available in reference 1. 
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Section III 

DEVELOPMENT OF THE OPERATIONAL SPECIFICATION 
FOR COMPUTER DIRECTED TRAINING 



3.1 INTRODUCTION 

The rationale for using computer directed training in Query 
Language was described in Section 1.4.2 of this report. The 
terminal product of this phase of the project is the Operational 
Specification for Computer Directed Training in Intermediate 
Query Language, Model 11^ for System 473L, February 195fo^. ^This 
operational specification completely describes the training de- 
sign and the necessary operating procedures for the proposed 
computer directed training capability. A description of the 
scope of programming required for the implementation of this 
capability, and the impact it will have on System 473L opera- 
tional capabilities is contained in a report prepared by the 
Federal Systems Division of International Business Machines, Inc.; 
this report is entitled Computer Directed Training; System 473L 
Query Language , April 1966**? The general conclusion stated in 
this document, as a result of programming analysis, is that the 
computer directed training capability would be compatible with 
System 473L equipment and prop;ramming subsystems, can be utilized 
in a manner similar to that of existing operational capabilities 
using an overlay, would have little impact on storage require- 
ments for data and programs, and would have no impact on the 
simultaneous utilization of existing operational capabilities. 

The training design and operating procedures were developed 
with specific reference to the Librascope 3055 computer and 
Intermediate Query Language, Model II, as they are intended for 
use in System 473L. However, the specific design described by 
the operational specification may be used as a model for the 
development of similar training programs in other command and 
control systems. 

3.2 DEVELOPMENT OF THE OPERATIONAL SPECIFICATION 

Each capability of System 473L is described in detail by an 
operational specification. The general contents and format of a 
System 473L operational specification are predefined. To ensure 
that the operational specification for the proposed training 
capability would meet approved 473L standards, and to provide 
guidance regarding the feasibility of proposed training features, 
the developmental process required close interaction between the 
A»I»R project staff and technical experts on System 473L from the 
Federal Systems Division of International Business Machines, Inc. 

^This document is reference 4 . 
^This document is reference 5* 
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The operational specification evolved through predeslgnated 
stages, which required the development, revision, and Integra- 
tion of three successive parts: the training design, or training 
sequence logic; the procedural flow diagram; and the specifica- 
tion of operating procedures. 

3.3 DESCRIPTION OF THE OPERATIONAL SPECIFICATION 

The following sections briefly describe the training design 
and the operating procedures for the computer directed training 
capability. More detailed Information Is available from reference 
4. 

3.3.1 The Training Sequence Logic 

3.3.1.1 Complexity of the training design . An Ideal design for 
computer directed training would permit maximum Interaction 
between the computer and the Individual trainee. Including com- 
plete analysis of responses and computer generation of all evalua- 
tions, directions, problems for Individual trainees, and responses 
to student-generated requests, with all communications In more or 
less unrestricted English. However, this Is not feasible for use 
with an operational system. In developing the training sequence 
logic, there was a need to limit the complexity of the training 
design In order to optimize the feasibility of Implementation In 
terms of a) the need to minimize any possible conflict with 
other System 473L operational capabilities; b) the cost of de- 
veloping a computer program to Implement the proposed logic; and, 
c) the cost of training Itself -- this would also Increase with 
Increasing complexity of the training design. 

The major training restrictions Imposed to Increase the 
feasibility of computer Implementation and reduce cost were: 

1) computer analysis of trainee responses on designated 
parts of QL statements, as opposed to analysis of com- 
plete QL statements. 

2) the use of "canned" answers (vs computer-generated 
answers) to problems requiring computer analysis, so 
that analysis of a trainee's response could be accom- 
plished by a simple matching process. 

3) the use of fixed formats for training materials and 
problems, so that content -- but not format -- changes 
could be made without a programming change. 

4) the use of only one major level of remedial training 
after each evaluative problem section. Thus, no 
remedial loop Is used to correct errors on a remedial 
sequence Itself. 
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5) the use of a limited number of evaluation ratings to 
evaluate a trainee *s performance and assign remedial 
materials. However, the error criteria for the 
various ratings can be stored in such a way that 
changes In these criteria would be relatively easy 
to make If experience so dictates. 

3.3.1.2 Provision for adaptation of the capability . Several 
provisions were made to permit adaptation of the capability for 
anticipated future changes In the training design and Query 
Language, and to permit training In an entirely different sub- 
ject. To achieve maximum flexibility without any programming 
change, the provisions for adaptation are through data mainte- 
nance. Some of the major areas In which data maintenance 
changes may be made are: the content of any cue -- i.e., the 
content for any unit of presentation to the trainee; the number 
of cues within a set; the criteria by which trainee errors are 
evaluated; and, the remedial continuations for trainee errors . 

3.3.1.3 The training sequence logic . The proposed computer 
directed training capability has three basic uses for training 
In Query Language, Model II: l) to train personnel In the use 
of Intermediate Query Language, Kodel II; 2) to provide perform- 
ance data for the trainees and supervisory personnel; and, 3) to 
provide proficiency maintenance training for trainees who have 
previously completed this or some other training program In 
Query Language, Model II. 

Salient features of the training sequence logic, designed 
to optimize training effectiveness and efficiency within the 
bounds of computer feasibility and cost. Include the following: 
presentation of training materials on the console display screen; 
computer evaluation of trainee performance based on the number 
and kind of errors and pre-stored evaluation ratings; computer 
determination of training areas requiring remedial work for 
each trainee; computer determination of the specific remedial 
sequence In each area that is appropriate to tl~ie kind and number 
of trainee errors; and, periodic px^ovlslon of optional remedial 
work for trainee selection. 

The training course designed for Initial use has a built-in 
research design on the relative effectiveness of forcing trainees 
to take remedial work appropriate to the number and kind of their 
errors, in addition to periodic provision of optional remedial 
work (called Research Group "A"), as opposed to the provision of 
optional remedial work only (called Research Group "b'*). This 
comparison would provide some of the information needed to make 
revisions In the training logic that would Improve the overall 
effectiveness and efficiency of the computer directed training 
course . 

Except for the remedial materials, whose presentation Is 
contingent on trainee errors and/or selection, the sequence of 
training materials Is fixed. The initial training sets provide 

33 



a basic orientation to the use of the instructional materials, the 
use of the console, the data files accessed by Query Language, the 
overall structure of Query Language statements, and the use of the 
Air Force Manuals that describe the general nature and contents of 
each data base file and the vocabulary used to reference the file 
contents. Subsequent sets provide extensive and cumulative 
training and evaluation on the use of Query Language to retrieve 
desired information from the data files. The Query Language 
topics covered by the computer directed training course are de- 
signed to progress in the order specified by Appendix D of this 
report . 

Any deviation from the main training path for the computer 
directed training course is contingent on the trainee's perform- 
ance and/or his own optional selection of remedial materials. 
The main training path and the remedial branches are shown in 
the Training Sequence Logic Flow Diagram, Figure 8. Explanatory 
notes for this logic on a general level are given in the para- 
graphs below. ^ 

The Training Sequence Logic Flow Diagram shows the main 
training sequence for all cues (individual units of presentation 
to the operator), points requiring a decision by the operator, 
points requiring computer evaluation of a trainee's performance, 
points requiring computer determination of any necessary branch- 
ing, all remedial traifling paths, and points at which data print- 
outs occur. 

To facilitate understanding of the Training Sequence Logic 
Flow Diagram, a condensed outline of the basic training sequence 
and remedial branches is presented below. 



Basic 

Training 

Sequence 



Remedial 

Branches 

for 

Excess 

Errors 



X SETS (5 sets) 


Y SETS (20 sets) 


LX-PX -- Eval 


LY-PY Eval-PKY*-Free Choice Review 


1 
1 
' 1 


*FEY not avail after Posttest PY 
1 



I 

A FIXED SEQ FROM 
LAST LX (ONE SEQ, 
REGARDLESS OF TYPE 
OF ERRORS) 



I 

ONE OR MORE REMEDIAL SEQUENCES 
APPROPRIATE TO l) THE PARTICULAR 
AREAS IN WHICH ERRORS WERE MADE 
AND 2) THE NUMBER OF ERRORS MADE 



Spor a complete explanation of all points on this diagram, the 
reader is referred to reference 4, the Operational Specification 
for Computer Directed Training in Intermediate Query Language , 
Model II, for System 473L J 
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NOTES: The cues used by this capability may be divided into two 
major classes s (l) the cues that provide a basic foundation for 
the course but do not teach Query Language; these cues are divided 
into sets called "x" SETS; and (2) the subsequent cues that cover 
all aspects of intermediate QL; these cues are divided into sets 
called "y" SETS. 

1) The X SETS COVER BASIC, NON-QL MATERIALS: 

a) the use of the instructional materials 

b) the use of the console and the CDT overlay 

c) a basic introduction to the files and structure of QL 

d) the use of the manuals covering the QL data base and 
system vocabulary 

2) The Y SETS cover all training and evaluation materials 
provided on the use of Query Language -- i.e., all mate- 
rials not covered by X SETS 

3) For any X SET: 

LX = training sequence -- a section within the X SET 
PX — series of problems on the LX -- a section within 

the X SET 
Eval = evaluation of errors on the PX section 

4) For any Y SET: 

LY = training sequence -- a section within the Y SET 

PY — series of problems on one or more of the preceding 
LY's -- a section within the Y SET 

Eval := evaluation of errors on each area of QL subjected 
to analysis by the last PY section 

PEY = series of frvee-form practice exercises (which the 
trainee answers with complete QL statements and on 
which he receives feedback). This is a section 
within each of the Y SETS except the last -- this 
is not available after the posttest 

Free-Choice Review = at end of each Y SET, the trainee 
has the option of taking review on any area sub- 
jected to analysis by the last PY section 

5) Throughout the entire program there are four general 
types of sets -- the basic, non-QL X SETS and three 
kinds of QL Y SETS. These sets are described below: 

a) 5 basic, non-QL X SETS 

b) 13 INDEPENDENT Y SETS for each of which the PY sec- 
tion tests only the materials covered by the LY sec- 
tion in the same set 

c) 6 CUMULATIVE Y SETS, used at appropriate points, in 
which the PY section tests all materials covered after 
the last CUMULATIVE Y SET 

d) one POSTTEST Y SET: this is the last Y SET, for which 
the LY section reviews all materials covered in the 
program and the PY section is the POSTTEST, which tests 
all the materials covered over all of the Y SETS in the 
program. 
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An understanding of the Training Sequence Logic Flow Diagram 
will also be enhanced by an understanding of cue types. There 
are two general types of cues used by this capability: 

1) Instructional cues. These cues are used primarily to 
instruct the operator (trainee) in the proper procedure for 
making a transition from one point in the training sequence to 
another. These instructional cues usually indicate the available 
options for continuing, and in some cases they provide feedback 

to the trainee regarding his performance on the last cue or series 
of cues. On the Training Sequence Logic Plow Diagram, most of the 
instructional cues are represented by a cue number, e.g., Ql, 
enclosed in a circle, and a brief description of the cue's func- 
tion. Instructional cues that provide an introduction to a 
sequence of learning cues are specified to the right of the 
bracket enclosing the series. For example, on page 1 of Figure 8, 
Q7 is an instructional cue that precedes a series of learning cues 
(which are, in this case, Q9's). 

2) Learning cues. These cues are sequenced so that the 
trainee will learn the desired criterion behaviors, e.g., how to 
write a SUM function. On the Training Sequence Logic Flow Dia- 
gram, learning cues are differentiated according to their train- 
ing function and according to their format. 

According to its training function, which may change from 
one part of the training sequence to another, a learning cue is 
identified as one of the following types: LX, PX, RX, LY, PY, 
FEY, Text, CUM TEXT, RY, CUT^ REVIEW, PRACTICE Y, and CUM PRACT 
PROB. For example, on page 1 of Figure 8, the first series of 
learning cues is represented by ( LX-1 ) 

According to its format, a learning cue is Identified as 
one of the following types: Q8, Q9, Qll, QI9, and QI9A. For 
example, on page 1 of Figure 8, the first series of learning 
cues is identified as a series of Q9'3 preceded by an instruc- 
tional cue (Q7). 

The different types of training functions for learning cues 
will now be explained. The learning cues used by this capability 
are logically and sequentially divided into two major groups: 

1) The non-QL learning cues. These cues provide a basic 
foundation for the course but do not teach Query Language Itself. 
They are divided into the sets called "x" SETS, which precede 
all other learning cue sets used by this capability. Thus, on 
page 1 of Figure o, the first series of learning cues have an X 
subscript, which identifies them as non-QL learning cues. 

2) The QL learning cues. These cues provide trainees with 
all training and evaluation materials that are needed in order to 
learn to use Intermediate Query Language, Model II; these cues 
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are divided into the sets called "y" SETS. Thus, on page 2 of 
Figure 8^ the first series of learning cues have a Y subscript, 
which identifies them as QL learning cues. 

In each non-QL X SET, learning cues may be further subgrouped 
according to their sequence and function. There are two major 
sections of learning cues in each X SET: 

1) An LX section. This is a sequence of cues (Q9*s) used 
primarily for training. Each LX section has two subsections: 
a) the initial series that provides basic training, and b) the 
final series that reviews the information taught in the first 
section . 

2) A PX section. This is a sequence of cues (Q8*s) used 
primarily for evaluation of the trainee's proficiency on the 
information covered in the preceding LX section. 

In each X SET^ these two sections are followed by computer 
error-analysis of the trainee's responses on the PX cues and, if 
necessary, appropriate remedial work . 

Learning cues may also be subgrouped in each Y SET according 
to their sequence and function. There are three major sections 
of learning cues in each Y SET: 

1) An LY section. This is a sequence of cues (Q9's) used 
primarily for training. Each LY section has three subsections, 
in this general order: a) the basic training sequence, called 
a TEXT sequence; b) the sequence that reviews the information 
taught by the TEXT sequence -- this is called a REVIEW sequence; 
and c) a series of cues that emphasize the development of QL 
statements for specified data retrieval problems. This series 
is intended to give the trainee practice in using the QL ele- 
ments taught in the TEXT and REVIEW sequences. This is called 

a PRACTICE PROBLEM sequence. 

2) A PY section. This is a series of cues (Qll's) used 
primarily for evaluation of the trainee's proficiency on the 
information in one or more of the preceding LY's. The com- 
plexity of the computer error-analysis on each problem is 
minimized by the restriction of error-analysis to specified 
parts of the appropriate QL statement « 

In each Y SET^ these two sections are followed by computer 
error-analysis of the trainee's responses on the PY cues and, if 
necessary, appropriate remedial work . There are three levels of 
remedial work, appropriate to three evaluation-ratings, GOOD, 
AVERAGE and POOR. In general, remedial work for a rating of 
POOR is excerpted from a TEXT subsection of an LY section; 
remedial work for a rating of AVERAGE is excerpted from a 
REVIEW subsection of an LY section; and, remedial work for a 
rating of GOOD is excerpted from a PRACTICE PROBLEM subsection 
of an LY section. Since remedial work in an area is forced on 
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a trainee in Experimental Group A if his FY errors on that area 
were excessive^ any remedial sequence taken at this point is 
called a FORCED REMEDIAL sequence. 

3) Following the error-analysis and any necessai*y remedial 
work for a Y SET is a PEY section. This is a series of cues 
(Q19's and Q19A's) that give the trainee practice in developing 
complete QL statements for specified problems^ with feedback to 
the trainee indicating the correct QL statement for each problem 
presented. This section is not used for computer evaluation; it 
is only used for a trainee's self -evaluation. Since the trainee's 
answers in this section are not evaluated by the computer, the 
answer format used by the trainee does not constitute a problem 
for error-analysis. Therefore, the trainee types the entire QL 
statement as his answer, not Just the restricted excerpts required 
for his answers in the PY section. Since these cues provide prac- 
tice in writing QL statements and the answer formats are not 
artificially restricted by the length of the answer required, 
these cues are called free-fonn practice exercises. 

For each Y SET, after these three sections are complete, 
the trainee is given the option of taking remedial work in one 
or more areas fcr which the trainee's responses in the last PY 
section were evaluated. Since remedial work at this point is 
taken only by free-choice and consists of REVIEW materials from 
one of the last LY sections, a remedial sequence that the trainee 
chooses to take at this point is called FREE-CHOICE REVIEW. As 
noted earlier, this is the only remedial work available to Experi- 
mental Group B. 

As mentioned previously, the X and Y SETS are different in 
terms of their position in the overall training sequence and their 
general training functions: X SETS are placed first and emphasize 
non-QL topics; Y SETS are last and emphasize the use of QL ele- 
ments. Cue sets may also be differentiated in terms of their 
general training function and the overall scope of their training- 
and-evaluation materials. In tei^ns of function and scope, there 
are four types of sets: 

All of the X SETS are of one type: 

1) Npn-QL PRETEST sets. These precede the QL materials 
that are tested by the final posttest; in each set, the RX sec- 
tion tests only the materials covered by the LX section in the 
same set. 

The Y SETS include three different types of sets: 

2) QL, INDEPENDENT sets. In each set, the LY section teaches 
new materials, and the PY section tests only the materials covered 
by the LY section in the same set. 
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3) QL, CUMULATIVE sets. In each set, the LY section reviews 
the materials covered in the INDEPENDENT sets that followed the 
last CUMULATIVE set, and the PY section tests all materials cov- 
ered after the last CUMULATIVE Y SET. 

4) The QL, POSTTEST set. In this set, the LY section reviews 
the materials covered throughout all of the sets. The PY section 
is the posttest, which tests all materials covered over all of 
the other Y SETS in the program. It is, in effect, the end-of- 
course criterion test. 

The entire program consists of the following sets, in the 
order described : 

1} five non-QL, PRETEST X SETS 

2) a series of 13 QL INDEPENDENT Y SETS, with six QL 
CUMULATIVE Y SETS Interpolated at appropriate points 
In the overall sequence 

3) one QL, POSTTEST Y SET. 
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The proposed program will provide evaluative data that are 
more extensive than that generally obtained by non-computer-based 
training studies. Such data will make It possible to relate vari- 
ous features of the program to performance measures. The evalua- 
tive data that will be made available consists of the following 
Items: 

1. Summary data for all trainees who have completed the 
program at any time 

a. the date 

b. the total number of trainees who have completed 
the program 

c. the total number of trainees still In progress 
on the program 

d. the average time to complete the program: for 
Experimental Group A and for Experimental Group B 

e. the average percent error on the program: for 
Experimental Group A and for Experimental Group B 

f . the average percent error on the posttest: (l) on 
the first try and (2) on the final (second) try: for 
Experimental Group A and for Experimental Group B 

g. the average percent error on each subcategory of QL: 
(l) averaged over all evaluative problems except the 
posttest for Experimental Group A and for Experimental 
Group B; and, (2) averaged over the posttest for all 
trainees . 

2. Individual trainee data for any trainee except a profi- 
ciency maintenance trainee, updated for any point during 
training and the final, updated data at the time the 
trainee completes the program 

a. the date on which the data is obtained 

b . the trainee ' s name, serial number, and experimental 
group (A or B) 

c. the cumulative training time for the trainee (at 
the time these data are obtained) 

d. the trainee's average percent error over all evalua- 
tive problem sections requiring QL compositions 
except for the posttest 

e. the trainee's average percent error on the posttest: 

(1) for the first try on each problem; and, (2) for 
the final (second) try on each problem 

f. the trainee's average percent error on each of the 
initial problem sections that do not require com- 
posing QL statements 

g. for each posttest problem: (1) the number of tries; 

(2) the possible number of errors; (3) the number of 
errors the trainee made on the first try; (4) the 
number of errors the trainee made on the second 
(final) try 

h, for each QL subcategory: (l) the average percent 
error for the trainee over all QL-evaluative problem 
sections except the posttest (i.e., during training 
on Query Language); (2) the average percent error 
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for the trainee on the posttest; (3) the number of 
times the trainee chose that subcategory for free- 
choice review over the entire program 
i. the trainee's sequence of remedial materials for 
the last 20 training sets. Identified as either 
forced remedial or free-choice and further Identified 
by the point at which these materials were taken 
(i.e.^ identified by the evaluative problem section 
immediately preceding each remedial set), the subcate- 
gory for which they were given^ and the trainee's 
rating on that subcategory. Since the trainee's use 
of any remedial sequence for one of the initial, non- 
QL problem sections can be determined by whether his 
percent error on that section exceeded 5a ^^^ since 
there is only one remedial sequence for each of the 
non-QL problem sections, the trainee's path through 
the X SETS is already determined. Since the basic 
(vs. remedial) training sequence of the Y SETS is 
known, the above information will complete the set of 
information needed to determine the trainee's entire 
path through the program, and his location at any time 
prior to completion. 

3. The complete QL statements typed by each trainee as 
answers to the data-retrieval problem in the practice 
exercises . 

4. The complete QL statements typed by each trainee as 
answers to the posttest problems and the parts of these 
statements, as typed by the trainee, that underwent error 
analysis . 

The comparison between Experimental Groups A and B plus an 
analysis of the various types of data made available, as described 
above, would not only provide information needed to make revisions 
in the training logic that would improve the overall efficiency 
and effectiveness of the computer directed training (CDT) program, 
but such information would also increase the general level of 
knowledge with respect to the potential role of computer-based 
instructional systems in the military environment. Thus, if the 
CDT program Is shown to be an effective teaching instrument, this 
computer directed training capability could be used advantageously 
for teaching the use of other System 473L capabilities and as a 
general model for application to training for other computer- 
based systems . 

3 .3 .2 Operating Procedures and the Procedural Flow Diagram . Any 
operator-processor communication for System U73L requires knowledge 
of the integrated console operating procedures . These are 
described In an Air Force document, 473L-OS-43: Operational 
Specification for the Integrated Console, Model II . In addition, 
each capability will employ a separate overlay that, when used with 
the integrated console process step keys, will present to the opera- 
tor the capability options. The function of each process step 
key associated with the overlay for the computer directed train- 
ing capability is shown in Table 6 of this report. 
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TABLE 6 

Design for Computer-Directed Training Overlay 
Process Step Key Functions (Page 1 of 3) 



Key Designation 
LI 



TRAINEE 



L2 



INSTRUCTOR 



L3 



PROFICIENCY 

MAINTENANCE 



L7 



SUMMARY 



l8 



INDIVIDUAL 

TRAINEE 



L9 



UPDATE 
TRAINING 
MATERIAL 



LIO 



FREE 
CHOICE 



Function 

This key signals the program that the operator 
wants to start or continue training on the CDT 
program. When this key is pressed, the Name 
and Serial Number Entry cue (Q6) appears on 
the display screen. 

This key signals the program that the operator 
wants to either obtain trainee data (for print- 
out or deletion) or to update training materials, 
When this key is pressed, the Directions for 
Obtaining Data or Updating cue (Q3) appears on 
the display screen. 

This key signals the program that the operator 
wants some of the CDT materials that are avail- 
able for proficiency maintenance. When this 
key is pressed, the Name and Serial Number Entity 
cue (Q6) appears on the display screen. 

This key signals the program that the operator 
wants P/O 1, the line printer printout of 
Summary Data for all trainees (except profi- 
ciency maintenance trainees). When this key 
is pressed, printout #1 will be printed out. 

This key signals the program that the operator 
wants to obtain data for an individual trainee, 
either on a printout or for deletion. When this 
key is pressed, the Directions for Obtaining or 
Deleting Data on an Individual Trainee cue (Q5) 
appears on the display screen. 

This key signals the program that the operator 
wants to update some of the training materials 
(i.e., the content of some of the cues). 



ignals the program that the operator 
ake remedial work in one or more areas, 
key is pressed, either the Index for 
e Review for Trainees Who Have Not 
the Program (Q2l) or the Index for 
e Review for Trainees Who Have Cora- 
Program (Q4) will appear on the dis- 
n, depending on whether the trainee 
ted the posttest or not. 



This 


key s 


wants to t 


When 


this 


Free- 


-Choic 


Completed 


Free- 


-Choic 


pleted the 


play 


scree 


has comple 
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TABLE 6 (cont. ) 
(Page 2 of 3) 

Key Designation 

Lll 



POSTTEST 



L12 



TRY 
AGAIN 



L13 



ERROR 

MATRIX 



L14 



EVALUATION 

CONTINUATION! 
MATRIX 



L15 



CUE 
FILE 

MATRIX 



Ll6 



CRITERIA 

MATRIX 



L17 



FLEX 
COURSE 



Function 

This key signals the program that a proficiency 
maintenance trainee wants to take the posttest 
(the very last FY section^ in the last set, Y = 
20). When this key is pressed, the first cue 
in the posttest (a Qll) will appear on the dis- 
play screen. 

This key signals the program that the opei^ator 
wants to take the last-presented problem again. 
When this key is pressed, the last pr'oblem in 
the FY section being worked on, which is a 
problem cue requiring computer error-analysis of 
the trainee's answer(s) (a Qll),will re-appear 
on display screen. 



This key signals the program that for updatin;3 
purposes, the instructor wants to obtain P/oS, 
the Possible Number of Errors on Each Subcate- 
gory for Each PY Section; on Each PX Section; 
and on Each Posttest Problem. When this key is 
pressed, printout #8 will be printed out on the 
line printer. 

This key signals the program that for updating 
purposes, the instructor wants to obtain P/07> 
Remedial Continuations. When this key is pressed, 
printout #7 will be printed out on the line 
printer. 

This key signals the program that for updating 
purposes, the instructor wants to obtain P/05j 
Cues (the contents for all cues). When this 
key is pressed, printout #5 v'ill be printed out 
on the line printer. 

This key signals the program that for updating 
purposes, the instructor wants to obtain P/o6, 
Criteria. When this key is pressed, printout 
#6 will be printed out on the line printer. 

This key signals the program that for updating 
purposes, the instructor wants to obtain P/OS s 
FLEX COURSE, which specifies the number of 
PRETEST (X) SETS, the number of POSTTEST prob- 
lems, and the number of subcategories . When 
this key is pressed, printout #9 will be printed 
out on the line printer. 
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TABLE 6 (cont. ) 
(Page 3 of 3) 

Key Designation 

L25 



CONTINUE 



L27 



SKIP 



L29 



TERMINATE 
TRAINING 



Function 

This key signals the program that either the 
operator does not wish to exercise any special 
training options available at this time or, 
none are available. When this key is pressed, 
the next cue that is available in queue (with- 
out branching out to pick up cues that are not 
presently stacked up) will appear on the dis- 
play screen. The type of cue that is presented 
will vary from one training point to another. 



This key signals the program 
does not wish to complete the 
on which he is now working, 
pressed, the trainee will be 
continuation point that he wo 
had he completed the ren.ed ial 
trainee is still in training 
and he has not yet taken the 
is a proficiency maintenance 
appear on the display screen; 
just completed the CDT progra 
posttest, Q23 will appear on 



that the operator 

remedial sequence 
When this key is 
sent to the same 
uld have reached 

sequence: if the 
on the CDT program 
posttest, or if he 
trainee, Q20 will 

if the trainee has 
m including the 
the display screen. 



This key signals the program that the trainee 
is going to stop work on the CDT program at the 
present time. When this key is pressed, Q32 
will appear on the display screen, and the 
storage of data for a trainee on the work done 
so far on a set will continue until storage is 
complete. 



A diagram of the training sequence in terms of the presenta- 
tion of specific cues and capability options is given in the 
Procedural Flow Diagram for the Computer Directed Training Capa- 
bility. This may be referenced in the Operational Specification 
for Computer Directed Training in Intermediate Que r;^^ Language , 
Model II, for System 473L ^o^ 



lU 



See reference 4. 
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3.4 DISCUSSION AND RECOMMENDATIONS 

The techniques for a computer clrected training program 
briefly described here^ and more fully described In the 
Operational Specification, are sufficiently general as to be 
applicable to other programs of training In artificial languages 
that are relatively fixed In format and vocabulary (such as 
computer languages ) . However, It Is recognized that other com- 
puter systems may not have available the type of equipment for 
which this program was designed. The only major feature of the 
Instructional system that may not be commonly available on other 
systems Is the overlay keyboard. The overlay system on the 
Integrated console was not originally designed to perform an 
Instructional function, but It provides an excellent opportunity 
for permitting limited control of the instructional process by 
the trainee and Increases the scope of interaction at the com- 
puter Interface . However, this overlay keyboard Is not essen- 
tial for a computer directed training capability. The functions 
performed with the overlay could be simulated by outputs on a 
display screen or typewriter providing a trainee with choices 
that would have been Indicated on the overlay. Although this 
method may be less economical, it would still be functional. 

It is felt that the major advantages of the CDT instruc- 
tional system are the periodic analysis of the trainee's input 
and the selection of appropriate remedial materials basec on 
this analysis, the extensive capabilities for output data, and 
the introduction of some learner control over the selection of 
training materials. The latter feature would be especially 
valuable for a proficiency maintenance training sequence, where 
individual needs would vary greatly. 

Proficiency maintenance Is a particularly critical training 
problem with command and control systems where there is a need 
for highly proficient personnel at all times but where there is 
little opportunity for actual practice in real or simulated 
situations. It would be possible to expand the computer clrected 
training course described here to provide frequent realistic 
combat exercj.ses and to provide, on the basis of an analysis of 
errors made, a series of training exercises uniquely designed 
to build proficiency In weak areas. This approach could also 
be generalized to other command and control systems where pro- 
ficiency must always be at a maximum but where actual practice 
may be at a minimum. 

Looking further Into the future. It would be possible to 
build Into a CDT program appropriate statistical tests that 
would determine the effectiveness of the program as measured 
against appropriate criteria of terminal performance. With 
Initial entrance data (e.g., initial aptitude, I^, etc.) and 
performance data collected at appropriate points throughout the 
program, statistical computations could be performed after an 
appropriate N of trainees had been obtained. The results of 
such an analysis may suggest changes in the program, e.g., a 
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new organization cf the materials and/or a shift in the 
parameters used for deciding en the frequency and nature of 
the correctional training sequences. Adjustments of this 
sort could be accomplished automatically and internally by 
the computer Itself. Such a training system would be truly 
adaptive in that it would adjust Its own training strategy 
based on the measured success of the prograT. oelng usee*:. 
For example, in the program described in this report, the 
two experimental groups (A and B) could be compared in an 
analysis of variance design and if one group was always 
inferior, that group could be discarded, or. If a significant 
Interaction was found between the experimental group and seme 
other variable, e.g., some background factor, a decision logic 
could be internally prepared for deciding who got the "A** con- 
dition and who got the "fl" condition. For example, perhaps it 
would be shown by the analysis that trainees with previous 
experience with computer languages should be given the progi'am 
allowing self choice (i.e., optional remedial sequences) > 
whereas those trainees with no previous experience in the 
field should be given the "lock-step" program (i.e., the pro- 
gram with forced remedial sequences). Entering trainees, in 
this case, would be asked to indicate this information and 
would be assigned by the computer to the appropriate program. 

Other relevant "questions*' of this kind could be "asked" 
in an effort to devise CDT systems that achieve a truly 
optimal relationship between level of proficiency and time In 
training. 1^ These approaches are within the present "state- 
of-art. They are expensive and they are yet to be proven by 
the test of practical experience. In spite of the fact that 
computer-directed training Is still In an experimental, neophyte 
phase, the military may provide the only feasible environment 
in which the effectiveness of such approaches can be demonstrated 
The existence and availability of large capacity computers, the 
number of men to be trained, the complexity of the subject 
matter, the difficulty of obtaining qualified instructors, and 
the critical nature of the training requirements, all exist in 
the context of military training as perhaps In no other sector 
of the training world. As has happened in so many other areas, 
a significant general advance In the technology of training 
could be initiated in support of military requirements. The 
impact of such an advance on both military and civilian training 
would be hard to overestimate. The Implementation of the CDT 
program described in this report would be an important step in 
this direction. Further refinements of the approach described 
here, applications to other military training problems, and 
inclusion of more sophisticated strategies such as have been 
suggested In this section of the report, would represent a pro- 
gram of both short- and long-term value to military and civilian 
training TT. 



■^■^See reference #6. 
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APPENDIX A 



Number of Frames in Each Volume of the Program'^ 



Volume 
Number 



Section of Program 



Number of 
Frames 



First 
Tryout 



Second 
Tryout 



I 

II 

III 

IV 
V 



VI 
VII 

VIII 
IX 

X 

XI 

XII 

XIII 

XIV 

XV 

XVI 

XVII 

XVIII 

XIX 

XX 

XXI 

XXII 

XXIII 



Preprogram 
General Orlen 
Organization 

Data Files 
Selection of 
Selection of 

Values Usln 

Manual 
Writing Slmpl 

The Simple 
Writing Slmpl 

Writing the 

Selector 
Writing Query 

Features of 
V/rltlng Query 

Features of 



tatlon 

of Information 



in the 



the Proper File Indicator 
the Correct Attributes and 
the Data File Contents 



e Query Language 
Qualifier 
e Query Language 
Output Director 



Statements - 

Statements - 
and Output 



Language Functions - Special 
the Qualifier 

Language Functions - Special 
the Selector 



GCD Functions 

MIN and MAX Functions 

GREATEST and LEAST 

Simple SU>'s 

More on SUM 

Titles 

Sorting 

Compound Qualifiers 

SAVE Procedures 

CHECK 

Review and Practice of II-XIX 

Use of the Data Control Manuals and 

Complex Queries 
Complex Queries - Computed Attributes 

and Special Uses 
Final Practice 



TOTAL NUMBER 



3^ 

133 

43 

21 



88 
55 

67 
64 



64 
137 

51 
113 

62 

18 
58 
46 
68 
8 

73 
98 

74 

17 

1448 



T^ 



There were no frames in Volumes XXIV through XXIX; these 
volumes contained reference or test materials only. 



APPENDIX B 
Final Test* 

ALL MANUALS, EXHIBITS, AND REVIEW PANELS ^iAY BE USED. 
A . General Information 

1. For what purpose Is the ^73L system designed? 

2. Determine what file Indicator you would use to obtain 
the following Information (you may use Exhibits 2N, 
3A and 3B): 

a. Specific data concerning current materiel Inventory 

b. The types of aircraft possessed by TAG 

c. Detailed characteristics of tactical aircraft 

d. The selection of a specific plan that fits certain 
requirements 

3. Label the following outputs with the appropriate output 
director. 

UNIT MDS AFLD NAME ANDREWS AMARILLO TRUAX 

23FIS F86A STATE MD TEX WIS 

F100i!' RNWY LENGTH 1200Qi 1350^ 90^0 

F104B 11000 10000 8500 

1 13F IS F 100B 9000 9500 7000 

F101G 8000 

F105D 

k. In the Query beloWj use brackets and labels to Indicate: 

a. The program Indicator 

b. The file Indicator 

c . The qualifier 

d. The output director 

e . The output selector 

f. The qualifier conjunction 

g. The output director conjunction 
h. Modifiers 

1 . Attributes 
j . Gomparators 
k. values 

RETRIEVE PLAN IDENT WITH PLAN GAT = EX, 

VUN PLAN IND = NO THEN PRINT PLAN IDENT, 

TYPE OP, START DATE 1 



-»t 



To save space, answer blanks are omitted in this reproduction 
of the test. 



APPENDIX B (cont .) 

5. Identify with arrows the points at which CHECK could be 
inserted in the following QL statement. 

RETRIEVE AIRFIELDS WITH COUNTRY = CANADA 

THEN PRINT AND COUNT AFLD NAME * AFLDS IN 

CANADA *n 

6. Match the following: 

AND a. commas between modifiers 

OR b. semicolons between modifier sets 

7 . Select the correct statements about SUM. 

a. Only one SUM expression may be used in a QL qualifier, 

b. Only one SUM expression may be used in a QL selector, 

c. If more than one SUM expression is used in a QL 
statement, all SUM expressions must have (or repre- 
sent) the same set of SUM control attributes. 

d. If more than one SUM expression is used in a QL 
statement, all SUM expressions must have (or repre- 
sent) the same set of summed attributes. 

e. A SUM function used in one subordinate query of a 
complex Query Language statement automatically applies 
to all subordinate Queries in the statement. 

f . A SUM function used in one modifier set of a compound 
qualifier automatically applies to all modifier sets 
of the compound qualifier. 

g. The only SUM expression that may be retained is SUM, 
referencing a SUM already computed in the qualifier. 

B. Problems 

Write the appropriate Query Language statements for the problems 
below (you may refer to any manuals or Exhibits). 

1. Write a Query to print out the entire UNIT/LOC STRNGTH file, 
sorted in descending alphabetic order according to the name 
of the installation. Specify also that the following SUMs 
are to be printed out: 

a. for each LSA, the number of U. S. citizens. 

b. for the entire file, the total number of U. S. citizens. 

2. You are checking the data files to determine whether some 
information has been omitted. Determine which planned 
operating bases have no values for ITEM in the files. 
Request a horizontal printout. 

3. Obtain the plan idents of all exercise plans that require 
CII9A aircraft and are scheduled to start on or between 

1 June 1966 and 15 July 1966. Use a horizontal display. 



APPENDIX B (cont.) 



Store a Query which will print out the number of authorized 
weapons systems for a force (UNIT, UNIT LOG, MDS and COMD) 
to be specified at a later tim.e. Leave room for other 
attributes in the selector. 



5. Write a Query to display the numbers of any SAVEd state- 
ments pertaining to F105G entries in FORCE STATUS. 

6. Make the necessary entries to complete this Query for 
Barclay Air Force Base. Add the title "Barclay Status 
Data" to the output. 

RETRIEVE MATERIEL STATUS WITH POB = ' 1 ' THEN PRINT ALL1 
/ENCOMPLETE QUERY - TO COMPLETE TYPE INSERT NUMBER AND = 
SYMBOL FOLLOWED BY: 

ADDITIONAL ATTRIBUTES 

ADDITIONAL VALUES 

OTHER PORTIONS OF QUERY STATEMENT AS DESIRED. 
START FOR EACH NUMBER IN STATEMENT OR ERASE COMPLETE 
INSERT NUMBER (E.G., '3'). TYPE EOM AFTER LAST INSERT. 
VCHECK PUNCTUATION AND FORMAT BEFORE PRESSING ENTER. 
(1 = • 

7. Write a Query to select either airfields in Canada that 
are no more than 500 nautical miles away from Hill and 
can land or depart (whichever is lower) at least 10 air- 
craft per hour under IFR conditions or airfields in the 
United States that are no more than 350 nautical miles 
from KANSAS CITY and have the above arrival/departure 
restrictions. Specify a horizontal printout of airfield 
names, the country, the GCD from both airfields, and the 
number of aircraft that can be landed or departed 
(labeled MIN ARR/DEP). 

8. Write a Query to print out the names and GCD's from 
Bailey of all airfields that are no further from Bailey 
than Peterson is from Wallace. Eliminate Bailey, 
Peterson, and Wallace from being selected. 

9. Write a Query to find the LOG ABSTRACTS annex to plan 
32111 and print out the annex plan ident and the abstract 
Also, a count of the number of and the plan idents of 
other plans using this annex is desired. Specify a 
printout . 

10. Write a Query to select the medical plan(s) with the 

fewest total number of patients per day to be moved from 
the collect bases. Specify a vertical display of the 
plan ident, the total number of patients per day to be 
moved, and all strategic data related to the route seg- 
ment requiring the fewest number of AME crews per day 
at the staging base. 



APPENDIX B (cont.) 

11. Write a Query to find the closest airfield to SEMHOI 
(excluding SEMHOI) that Is a FOB for at least 10,000 
pounds of gaseous nitrogen. Obtain a vertical display 
of the airfield name, the GCD from SEMHOI, and the 
amount of gaseous nitrogen Intended for use there. 

12. Write a Query to find those FORCE STATUS entries with 
minimum ready weapons systems of at least 10. Entitle 
the minimum RDY WP SYS. Specify a vertical display of 
the force and the weapons system. 

13. Write a Query to find the total amount of jet fuel stored 
at Parkersburg (regardless of the POB of the fuel) and 
those airfields In logistics subarea 3A capable of storing 
at least that much Jet fuel. Specify a horizontal listing 
of the amount of jet fuel stored at Parkersburg, the names 
of the qualifying airfields, their GCD's from Parkersburg, 
and their Jet fuel capacities. 

14. Write a Query tc display the number of airfields In the 
United States having all navigational aids either avail- 
able or estimated and having at least 1 of the lighting 
facilities either available or estimated. 

15* Write a Query to find all plan Idents, the FOB's, the 

MDS's, and the number of airmen denloyed from a POB and 
MDS set (such as F105G's at Bailey) when at least 1000 
airmen are to be deployed from a set and the POB is in 
the United States or when at least 1000 men are to be 
deployed from a set and the airmen are from an Y(FG unit. 
Specify a sort in ascending order according to MDS. 

16. Write a Query to find the minimum authorized weapons 

system of E52A's for the 8lBW unit of TAG at Parkersburg 
(entitled WPSYS AUTH) and those plans which require no 
more crewed B52A's than the authorized weapons system. 
Specify a horizontal display of the weapons system, anc"* 
the plan Idents of qualifying plans. 



APPENDIX C 
Final Test Scoring and Answer Key 

Scoring of QL Statements on the posttest is based on: 

1. Program director — 1 point 

2. File indicator — 1 point 

3. WITH ~ 1 point 

h. Each modifier — 1 point 

5. THEN ~ 1 point 

6. Output director — 1 point 

7. Each phrase in selector* — 1 point 

8. Punctuation and spelling of entire QL statement — 1 point 

* A phrase may consist of a single attribute, a computed attribute, an 
attribute = INCR/DECR, or a SAVE instruction. For^xample, 

Q ^Z.^!^ Q) , ^^_ 

...THEN LIST AND COUNT^LD NAME, GCD, GCD (BEALE), (^ 



:OUNT^I 



'^UM AND TOTAL BY AFLD NAME (RNWY LENGTH = INCRTT ^^IN (^AX IFR ARR, 

MAX IFR DEP), TITLE = MIN TRAFF, ^CALIFORNIA AIRFIELD INFO*^ 

'^"SAVE BY SMITH 1966 "^ 

would be worth 12 points. 

Since most Query Language problems can be solved in more than one way, 
it is important that the test be scored by someone knowledgeable in Query 
Language. Each Query should first be assigned a number of possible points , 
according to how the student attempted to solve the problem, and then points 
subtracted for the items missed. For exgtmple, if a student wrote the Query 

RETRIEVE AIRFIELDS WITH AFLD NAME = PETERSON THEN RETAIN GCD (WALLACE): 
RETRIEVE AIRFIELDS WITH AFLD NAME i BAILEY AND PETERSON AND WALLACE, 
GCD (BAILEY<[R1, GCD (WALLACE), OR] THEN PRINT AFLD NAME, GCD (BAILEY)n 

for question number 8, he would receive IT possible points, and lose two: 

l) for using AND's in the AFLD NAME i' modifier, and 2) he would lose his 

punctuation and spelling point for using parentheses around WALLACE in the 

value [Rl, GCD WALLACE, OR]. 

It will be necessary, for each student, to total the possible points and 

the correct points and compute his percentage. 
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TEST MSWER KEY 



A. General Information 

1. provide information needed for decision-making 

2. a. MATERIEL STATUS 

b. FORCE STATUS 

c. TACTICAL CHAR 

d. PLAN IDENT 

3. LISTH OR PRINTH OR, HORIZONTAL 
LISTV OR PRINTV OR, VERTICAL 

k. In the Query below, use brackets and labels to indicate: 



a. 


The program indicator 


b. 


The file indicator 


c. 


The qualifier 


d. 


The output director 


e. 


The output selector 


f. 


The qualifier conjunction 


g- 


The output director conjunction 


h. 


Modifiers 


i. 


Attributes 


J. 


Comparators 


k. 


Values 



(a) 

program 
indicator 



RETRIEVE 




PLAN IDENT 

Thl 



(f) 

qualifier 
conjunction 

WITH 



Points 
1 
1 

1 
1 
1 
1 
1 



11 



(c) 
quaO^fier 



(h) 
modifier 



attribute comparator value 



PLAN 



CAT 



EX, 



modifier 



^ 



attribute ( i ) comparator ( j )value(k) 
VUN PLAN IND = NO 



output director output 
con junction ( g ) director ( d ) 



THEN 



PRINT 



output selector(e) 



attribute (i) 
PLAN IDENT, 



attribute (i) 



TYPE OP, 



attribute (^> 
START DATE *1 



5 . RETRIEVE AIRFIELDS WITH COUNTRY 



CANDA THEN 



PRINT AND COUNT AFLD NAME *AFLDS IN CANADA* ^ T 6 



APPENDIX C ( cont . ) 

Points 

6. a 1 
b 1 

7. (a?) 1 

b. 1 

CS) 1 



CD 



1 
1 
1 
1 



Problems 

1. RETRIEVE UNIT/LOC STRNGTH THEN PRINT INSTALLATION = DECR, ALL, 
SUIA AND TOTAL BY LSA (TOTAL US CIVILS)^ 8 

2. RETRIEVE MATERIEL STATUS WITH ITEM = BLANK THEN PRINTH POB -n 8 

3. RETRIEVE PLAN IDENT WITH PLAN CAT = EX, MDS = C119A, START 10 
DATE>01JUN66 and < 1^UL66 then LISTH plan IDENT -1 

k. RETRIEVE FORCE STATUS WITH UNIT = ?, UNIT LOC = ? , MDS = ?, 13 
COMD = ? THEN PRINT MIN (CRWS AUTH , ACFT AUTH) , ? SAVE -i 

5. FIND FORCE STATUS, F105G -t 1* 

6. RETRIEVE MATERIEL STATUS WITH POB = '1' THEN PRINT '2* n 2 
(1 = 'BARCLAY 2 = 'ALL ^BARCLAY STATUS DATA*' -n 2 

7. RETRIEVE AIRFIELDS WITH COUNTRY = CANADA, GCD (HILL <500), l8 
MIN (MAX IFR ARR, MAX IFR DEP) > 10, TITLE = MIN ARR/DEP; 
COUNTRY = USA, MIN (MAX IFR ARR, MAX IFR DEP) > 10, TITLE = 

MIN ARR/DEP, GCD (KANSAS CITY < 350) THEN PRINTH AFLD NAME, 
COUNTRY, GCD, MIN ARR/DEP i (OR PRINTH AFLD NAME, SAME n) 

8. RETRIEVE AIRFIELDS WITH AFLD NAI4E ?^ BAILEY OR PETERSON OR 9 
WALLACE, GCD (BAILEY < GCD (PETERSON, WALLACE)) THEN PRINT 

AFLD NAME, GCD ^ 



APPENDIX C (cont.) 



Points 



9. RETRIEVE PLAN IDENT WITH PLAN IDENT = 32111 THEN RETAIN 2k 
LOG ABST IDENT: RETRIEVE LOG ABSTRACTS WITH PLAN IDENT = 
[Rl, LOG ABST IDENT, OR] THEN PRINT PLAN IDENT, LOG ABSTR: 
RETRIEVE PLAN IDENT WITH LOG ABST IDENT = [R1, LOG ABST 
IDENT, OR] THEN PRINT AND COUNT PLAN IDENT n 

10. RETRIEVE MED PLAN RQMT WITH SUM BY PLAN IDENT (COLLECT CSU 10 
RQMT = MIN), STAGE AME RQMT = LEAST THEN LISTV SUM, RTE ID = 
GROUP -I 

11. RETRIEVE MATERIEL STATUS WITH POB ^ SEMHOI, ITEM = NITRO GAS, 28 
TOTAL ASSETS > 10000 THEN RETAIN POB: RETRIEVE AIRFIELDS WITH 
AFLD NAME = [R1, POB, OR], GCD ( SEMHOI = MIN) THEN RETAIN AND 
LISTV AFLD NAME, GCD: RETRIEVE MATERIEL STATUS WITH ITEM = 
NITRO GAS, POB = [R2, AFLD NAME, OR] THEN LISTV TOTAL ASSETS ^ 

12. RETRIEVE FORCE STATUS WITH MIN (CRWS RDY, ACFT RDY) > 10, TITLE 13 
= RDY WP SYS THEN LISTV COMD, UNIT, UNIT LOC, MDS, RDY WP SYSn 

13. RETRIEVE MATERIEL STATUS WITH ITEM = JET FUEL, STORAGE LOC = 20 
PARKERSBURG THEN RETAIN AND LISTH SUM (SL ASSET): RETRIEVE 
AIRFIELDS WITH LSA = 3A, JET FUEL CAP> [Rl, SUM SL ASSET, AND] 
THEN LISTH AFLD NAME, GCD (PARKERSBURG), JET FUEL CAP n 

lU. RETRIEVE AIRFIELDS WITH COUNTRY = USA, NAVAID FACILITY = TOWER 9 
AND APPROACH AND VOR AND TACAN AND VORTAC AND RBN AND AIR/GND 
AND DF AND GCA AND ILS, LTG FACILITY = ANY THEN COUNT 1 

15. RETRIEVE PERS DEPLOY STAT WITH SUM BY MDS BY POB (TOTAL ORG-AMN 11 

>1000), CRTY/STATE = USA; ORG KIND = RFG THEN LIST SUM = INCR, 

PLAN IDENT n 

16. RETRIEVE FORCE STATUS WITH COMD = TAC, UNIT = 8lBW, MDS = B52A, 21 
UNIT LOC = PARKERSBURG THEN RETAIN AND LISTH MIN (ACFT AUTH, 

CRWS AUTH), TITLE = WPSYS AUTH: RETRIEVE PLAN IDENT WITH MDS = 
B52A, CRWD ACFT RQ < [R1, WPSYS AUTH, AND] THEN LISTH PLAN IDENT"! 



APPENDIX D 
Sequence and Contents of the Computer Directed Training Sets 



X SETS 



INDEP 
Y SETS 



CUM Y SET 



SET CONTENTS 

A A brief introduction to the use of the console 

and the CDT overlay; use of the Instructional 
materials; use of the CDT Exhibit Book. 

B More detailed coverage on the use of the console 
and the CDT overlay. 

C The computer component and the input and output 
devices of the 473L System; the file struc- 
ture and the types of data storage utilized 
by System U73L; basic methods of data 
retrieval with System U73L; more on the use 
of the console and the CDT overlay. 

D Basic elements of a QL statement; introduction 
to Error Messages; use of the AP manuals 
covering the QL data base and system vocabu- 
lary; more on the use of the CDT Exhibit Book, 

E More on the use of the AF manuals covering the 
QL data base and system vocabulary; more on 
the use of the CDT Exhibit Book; more on the 
use of the console and the CDT overlay. 

F The Simple Qualifier 

Q Basic Instruction on writing the Output Director 
and Output Selector 

H Review of P and G 



INDEP 
Y SETS 



Special features of the qualifier 
Special features of the selector 



CUM Y SET 



Review of I and J 



INDEP 
Y SETS 



GCD functions 

MIN and MAX functions 



CUM Y SET 



Review of L and M 



INDEP 
Y SETS 



More on MIN and MAX functions 

GREATEST and LEAST^ and Review of MIN and MAX 



CUM Y SET 



Review of and P 



INDEP 
Y SETS 



INDEP 
Y SETS 



CUM Y SET ../T 

I" 

••<v 

CUM Y SET ,,iW 
INDEP Y SET ..ix 



POSTTEST 
Y SET 



Simple uses of SUM 

More complex uses of SUM 

Review of R and S 

Titles and Sorting and Compound Qualifiers 
SAVE Procedures and CHECK 

Review of U and V 

Complex Queries 

Review and Practice of Sets F through X^ 
and the Posttest 
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